Repository navigation
Add projection to FilterExec to avoid unecessary output creation #5436
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently FilterExec filters/outputs all columns present in the filter expression. This does unnecessary work, as we might not need certain columns later.
Describe the solution you'd like
Add projection to Filter/FilterExec and use it before passing the RecordBatch to the arrow kernel.
This saves a bit of computation, especially when there is only one column in the projection (fast case).
Also update the schema based on the projection.
Describe alternatives you've considered
Additional context
Reacted by Raphael Taylor-Davies
Activity
jackwener commentedon Mar 1, 2023
It's a great idea.
After it's finished, we also should do optimization job in
EliminateProjectionPushdownProjectionDandandan commentedon Mar 1, 2023
Thanks @jackwener yes, I agree (suggestion was actually inspired by your recent PR, thanks for that).
Filtershould also update the output schema based on projection, soEliminateProjectionshould be able to use the schema based on the change there.[-]Add projection to `FilterExec`[/-][+]Add projection to `FilterExec` to avoid unecessary output creation[/+]