Repository navigation
[EPIC]: Make PiecewiseMergeJoin work in Datafusion #17427
Description
Is your feature request related to a problem or challenge?
I will organize the following tasks that will be done in separate pull requests as #16660 got a little large and confusing.
The following is to be done roughly in order:
- feat:
ClassicJoinfor PWMJ #17482 - feat: Add Semi/Anti join to PiecewiseMergeJoin #18392
- Improve metrics accounting
- Mark join support
- Serialization
Smaller Tasks Along the Way:
- Move predicate extraction to Optimizer rule
- Support leftover predicates
- Support swap inputs
- Check
PiecewiseMergeJoinperformance for large tables #18221
Optimizations for semi/anti joins
- look into inner joins having the smaller side on the buffer side.
- Instead do no sort on buffer side, and just run a filter with the min/max value calculated from the stream side
The classic join and existence join will have their own stream implementation as the logic is completely different. Nice little suggestion from @2010YOUY01 😄
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Reacted by Yongting You and kamille
Activity
jonathanc-n commentedon Sep 5, 2025
This should be quite straightforward since the code is already done. Hopefully we can get started on IE joins after this
jonathanc-n commentedon Sep 5, 2025
take
2010YOUY01 commentedon Oct 16, 2025
Here is the background if anyone is interested:
(see top comment in
exec.rs) https://github.com/apache/datafusion/pull/17482/files#diff-bc8796293bb6ee5f3f2b43bfb2b43959726f0b458dd3475dbbb4ab6ea0fc5a8cfeat: `ClassicJoin` for PWMJ (#17482)
feat: `ClassicJoin` for PWMJ (apache#17482)
feat: `ClassicJoin` for PWMJ (apache#17482)
SubhamSinghal commentedon Jul 19, 2026
Hi @jonathanc-n — I'd like to help move the existence-join items forward for
PiecewiseMergeJoin. Are you still planning to pick up #18392, or is it open for someone else to take? Happy to defer if you're already on it.jonathanc-n commentedon Jul 20, 2026
@SubhamSinghal Yes feel free to take over!
feat(pwmj): support LeftSemi/LeftAnti existence joins via classic scan (
bench: pwmj left semi/anti join (apache#24160)
feat(pwmj): support LeftSemi/LeftAnti existence joins via classic scan (
bench: pwmj left semi/anti join (apache#24160)
bench: pwmj right semi/anti existence joins (apache#24458)
feat: proto serialization for PiecewiseMergeJoinExec (apache#24378)
bench: add PWMJ vs NestedLoopJoin criterion coverage for LeftMark/Rig… (
feat(pwmj): support RightSemi/RightAnti existence joins (apache#24457)
feat(pwmj): support left/right mark joins (apache#25467)
perf(pwmj):settle never-matching streamed rows against the buffered e…
feat(pwmj): support left/right mark joins (apache#25467)
perf(pwmj):settle never-matching streamed rows against the buffered e…
bench: classic pwmj sql benchmark (apache#25839)
perf(pwmj): build output batches from buffered runs and one streamed … (