Repository navigation
Physical plan pushdown for volatile predicates #16545
Description
Describe the bug
This is a follow-up to a discussion in #16325 (comment), which is not directly related to table sampling but could affect it.
I'd like to double-check if a volatile filter pushdown to a Parquet executor is expected. I had implemented the disabling of volatile pushdown filters for a logical plan in #13268. But it seems like the physical optimiser still pushes this predicate to an executor. Should we implement a similar mechanism to make volatile predicates as unsupported filters? In a current physical plan implementation, there is a concept of "unsupported" filters, which can be easily reused for it.
Current behaviour:
Before:
[2025-06-18T18:20:07Z TRACE datafusion::physical_planner] Optimized physical plan by LimitedDistinctAggregation:
OutputRequirementExec
ProjectionExec: expr=[count(Int64(1))@0 as count(*)]
AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))]
AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))]
FilterExec: random() < 0.1
DataSourceExec: file_groups={1 group: [[sample.parquet]]}, file_type=parquet
After:
[2025-06-18T18:20:07Z TRACE datafusion::physical_planner] Optimized physical plan by FilterPushdown:
OutputRequirementExec
ProjectionExec: expr=[count(Int64(1))@0 as count(*)]
AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))]
AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))]
DataSourceExec: file_groups={1 group: [[sample.parquet]]}, file_type=parquet, predicate=random() < 0.1
To Reproduce
set datafusion.execution.parquet.pushdown_filters=true;
create external table data stored as parquet location 'sample.parquet';
SELECT count(*) FROM data WHERE random() < 0.1;Expected behavior
I expect the physical plan optimiser doesn't perform pushdown of volatile predicates.
Additional context
No response
Activity
theirix commentedon Jul 2, 2025
@findepi could you please tell if this behaviour makes sense? If not, I could try fixing the physical plan as in #13268, where you have a review
theirix commentedon Jul 19, 2025
@alamb @adriangb, given your most recent work on pushdowns, does it appear to be a bug?
alamb commentedon Jul 21, 2025
I am not sure -- does this result in wrong results?
It does make sense in general to have the physical pushdown optimizer follow the same behavior as the logical optimizer
theirix commentedon Jul 22, 2025
We don't observe incorrect results (as was the case with the logical optimiser bug when evaluating a predicate twice).
It can produce skewed or non-reproducible results if a volatile predicate evaluates to different values for different row groups (I can create a test to reproduce it).
It could probably affect the table sampling implementation #16325 , which relies on inserting a volatile filter.
Yes, it is a reasonable assumption about an optimiser.
adriangb commentedon Jul 22, 2025
It seems reasonable to me to blacklist some filters. But given that users can create arbitrary trait implementations for PhysicalExpr we obviously can't black list everything. Unless there a volatility method on the trait?
theirix commentedon Jul 30, 2025
Thank you for the improvement, @adriangb !