Repository navigation
More accurate memory accounting in external sort #14748
Description
Is your feature request related to a problem or challenge?
#14644 fixed an external sorting bug. Each batch's memory overhead is estimated as 2 * batch memory size for the extra row conversion overhead. It works for common cases but still can fail. See #14644 (comment) and #14644 (comment).
To implement a more accurate memory size estimation for those edge cases, we can first do col->row conversion and measure the memory consumption. (depends on #7053)
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Reacted by Andrew Lamb
Activity
2010YOUY01 commentedon Mar 5, 2025
There is a small optimization can be done after we have accurate memory accounting #15017 (comment)
alamb commentedon Mar 12, 2025
I wonder if this is ready to do now?
2010YOUY01 commentedon Mar 13, 2025
Yes, I think so. However, this might cause a temporary slowdown due to an extra
row <-> colconversion. Since the performance can be brought back by reusing converted rows between operators, and also I believe correctness is more important, this is not a hard blocker.I plan to work on this ticket in the future, but if anyone is available to take it now, please feel free.
rluvaton commentedon May 23, 2025
This PR actually fix memory accounting: #15700
Just need to answer the comments and add tests to prove that the memory accounting is consistent
ding-young commentedon Aug 4, 2025
take
alamb commentedon Jun 4, 2026
I filed an epic with a bunch of related memory items to try and coordinate some response