You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Tune nested take compaction threshold with a query-level benchmark #7145
list_extract (#6498) and map_extract (#6845) share one policy for compacting nested take results: a buffer is compacted when its capacity exceeds 4× its used bytes plus 64. So far the threshold has only been measured at the kernel level. For 8,192 map<int,array<int>> rows with two 128-element values and about 75% missing keys, compaction makes m[1] 29–32% slower, while retained output capacity drops from 4,228,100 to 1,079,300 bytes (review). The kernel numbers miss the downstream effect: buffering operators such as sort reserve memory by capacity, so oversized output can raise memory pressure and spill earlier.
Describe the potential solution
Add a query-level benchmark that feeds nested m[k], element_at and arr[i] results into a buffering operator such as sort, and measures CPU time, peak memory and spills with ample and constrained memory. Use it to tune the shared threshold in nested_compaction.rs, or keep the current one if the results support it.
Activity