Repository navigation
Add documentation for support for skipping Parquet row groups #825
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We sometimes get questions about support for skipping Parquet row groups based on statistics. It seems that we do not have good documentation around this really cool feature, so we should write something up. We can base it on this response copied from the slack channel.
DataFusion has support for skipping entire row groups using predicates and min and max statistics.
It does not (yet) push the predicates down into the actual scan (e.g. to avoid materializing data that wouldn’t pass the predicate) — instead any row groups that are not pruned are decompressed into RecordBatches and then filtered.
Also, DataFusion will do “projection pushdown” — aka it will read only those columns needed to answer the query.
Describe the solution you'd like
Promote this cool feature in the documentation somewhere (user guide? README?)
Describe alternatives you've considered
None
Additional context
None
Activity
houqp commentedon Aug 6, 2021
seems like something that would be a good fit for design doc or user guide.
matthewmturner commentedon Sep 4, 2021
Hi there - I can work on this. Just to make sure I understand - would doing this at the scan level mean extracting the min-max from the compressed data in order to determine whether the row group even needs to be materialized?
With regards to the actual docs - does it make sense to add a general section on the main docs page to list the optimizations that are currently implemented / planned? i.e. whats here https://docs.rs/datafusion/5.0.0/datafusion/optimizer/optimizer/trait.OptimizerRule.html could be used for whats implemented.
tustvold commentedon Oct 20, 2022
FWIW this may overlap with #3464
alamb commentedon Nov 30, 2022
as of #4427 the existence of row group pruning will be present in the config settings
Also DataFusion now can push predicates down into the scan!
alamb commentedon Sep 18, 2024
The feature list is now pretty well listed here: https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/parquet/struct.ParquetExec.html#features
Maybe we can just add a link to the main page / docs site
alamb commentedon Feb 13, 2025
Proposed to add in
chore: Add more unsupported cases to supportedSortType (apache#825)
Add PyCapsule support for Arrow import and export (apache#825)