Repository navigation
Allow user defined ExecutionPlans to benefit from pushdown behavior #8096
Description
Is your feature request related to a problem or challenge?
#8073 adds support for projection pushdown in the physical optimizer.
However, its implementation directly checks for and special cases the built in ExecutionPlan nodes in DataFusion (which other optimizer passes do as well)
For example
} else if let Some(output_req) = input.downcast_ref::<OutputRequirementExec>() {
try_swapping_with_output_req(projection, output_req)?
} else if input.is::<CoalescePartitionsExec>() {
try_swapping_with_coalesce_partitions(projection)?
} else if let Some(filter) = input.downcast_ref::<FilterExec>() {
try_swapping_with_filter(projection, filter)?
} else if let Some(repartition) = input.downcast_ref::<RepartitionExec>() {
try_swapping_with_repartition(projection, repartition)?
} else if let Some(sort) = input.downcast_ref::<SortExec>() {
try_swapping_with_sort(projection, sort)?Describe the solution you'd like
It would be nice if there was an abstraction somehow where ProjectionPushdown could be used by extension / user defined ExecutionPlan nodes
Describe alternatives you've considered
There is a really nice discussion between @crepererum and @ozankabak here #8073 (comment) that provides insight and various options.
Additional context
No response
Activity
yyy1000 commentedon Jan 31, 2024
I'd like to help it.
A question is when I looked at #8073 (comment) , I understand adding a function to
ExecutionPlanand let everyimpl ExecutionPlanalso has it. But I couldn't understand what doesD: abstractionmean. 🤔berkaysynnada commentedon Jan 31, 2024
Hi @yyy1000, thank you for your willingness to help. I've been working on improving the projection optimizer rule to ensure the most efficient use of projections while minimizing table loads. I plan to complete my work in a few days and will open the PR then. That new rule will have a similar limitations, so I will seek suggestions on how to integrate the rule with the
ExecutionPlanandPhysicalExprAPIs. I believe your assistance will be much more valuable then.jayzhan211 commentedon Oct 24, 2026
The projection part of this is done:
ProjectionPushdownnow works throughtry_swapping_with_projection(#14235). The wider problem is still here, and it turns out to be about correctness as well as missed optimizations, so I opened #26697 to track it. Would it make sense to close this one in favor of that?alamb commentedon Oct 24, 2026
Makes sense to me