Repository navigation
Integrate BatchCoalescer into HashJoinExec and remove from CoalesceBatches optimization rule #18781
Description
Is your feature request related to a problem or challenge?
The LimitedBatchCoalescer can now be used to integrate batch coalescing directly in ExecutionPlan implementations, which is better than using an external operator for the reasons described on #18779
Currently HashJoinExec needs a CoalesceBatches operator after for performance, see
We should integrate coalescing directly in the operator
Describe the solution you'd like
- Add batch coalescing into HashJoinExec
- Remove HashJoinExec from the
CoalesceBatchesoptimizer rule:
Describe alternatives you've considered
@Dandandan did this for FilterExec in these two PRs, so we can probably follow a similar model
- Coalesce batches inside FilterExec #18604
- Remove FilterExec from CoalesceBatches optimization rule, add fetch support #18630
Additional context
No response
Activity
alamb commentedon Nov 17, 2025
FYI @gene-bordegaray this is one of the tickets I mentioned to you today
gene-bordegaray commentedon Nov 17, 2025
Ok great, I will spend some time reading up on the epic and see if I can pick up some of this work
jonathanc-n commentedon Nov 18, 2025
@gene-bordegaray Is it fine if I take over this one? I'm quite interested in the hash join stuff. I believe the repartition operator should be somewhat similar to this if you are interested
gene-bordegaray commentedon Nov 18, 2025
Yes yes, of course. I am just trying to learn so will be snooping 😅
jonathanc-n commentedon Nov 20, 2025
take
Dandandan commentedon Nov 28, 2025
FYI @jonathanc-n FYI I went ahead with it to do some performance experiments #18972
jonathanc-n commentedon Nov 28, 2025
@Dandandan Sounds good, sorry about that. Been a bit busy lately
Coalesce batches inside hash join, reuse indices buffer (#18972)