Repository navigation
parquet page level skipping (page index pruning) doesn't work with evolved schemas #5104
Description
Describe the bug
parquet page level skipping (page index pruning) panic's with evolved schemas
To Reproduce
Enable setting like
pub enable_page_index: bool, default = trueAnd then run test:
cargo test -p datafusion --lib -- evolved_schema_disjoint_schema_filterWhich fails
test physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter ... FAILED
failures:
---- physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter' panicked at 'index out of bounds: the len is 1 but the index is 1', /Users/alamb/.cargo/registry/src/github.com-1ecc6299db9ec823/parquet-31.0.0/src/file/metadata.rs:251:10
Expected behavior
Test should pass
Additional context
I found this while working to enable this feature by default
#5099
Activity
alamb commentedon Jan 29, 2023
FYI @Ted-Jiang -- I plan to fix this
alamb commentedon Jan 29, 2023
I am pretty sure the issue is that the page pruning is trying to use a column index into the merged schema to find the relevant statistics, but each file schema can be slightly different. I think the fix will be to look up statistics based on name rather than index
alamb commentedon Feb 12, 2023
The tests that fail are: