You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Limit the max merge degree during re-spill in external sort #16908
#15700 introduces a comprehensive approach for external sorts: if it's not possible to do the merge for all spills in one pass, it will merge as many spills as possible, do a re-spill, and iterate until we can produce the final output at once.
Note it is also use estimations for the memory accounting, under extreme edge cases this estimation can be inaccurate, causing the actual memory usage larger than the configured memory limit.
To prevent such unintentional cases and to enable performance tuning (merging a smaller number of batches at once is more cache-friendly/faster), we can introduce a additional configuration option: max_merge_degree: Maximum number of streams to merge during re-spills in external sorts
Activity
ding-young commentedon Aug 8, 2025
take