You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add MemoryReservation to batch splitting in joins #13003
In #12633@mhilton noted that joins sometimes generate giant record batches which causes issues. @alihan-synnada fixed this in #12969 but internally sometimes the joins still generate giant output batches.
Unfortunately this doesn't address the actual problem with creating giant batches, which is they require a lot of memory and that memory isn't accounted for in any MemoryPool. Wiring a MemoryReservation into BatchSplitter would probably be enough to address this though.
Describe the solution you'd like
I would like the memory accounting to take into account the large output batch
Describe alternatives you've considered
Wiring a MemoryReservation into BatchSplitter would probably be enough to address
Activity
jatin510 commentedon Oct 18, 2024
can i work on this task @alamb ?
alamb commentedon Oct 20, 2024
@jatin510 of course -- see the guide here https://datafusion.apache.org/contributor-guide/index.html#open-contribution-and-assigning-tickets !
jatin510 commentedon Oct 24, 2024
take