Skip to content

Tune nested take compaction threshold with a query-level benchmark #7145

Open

Description

@rich7420

What is the problem the feature request solves?

list_extract (#6498) and map_extract (#6845) share one policy for compacting nested take results: a buffer is compacted when its capacity exceeds 4× its used bytes plus 64. So far the threshold has only been measured at the kernel level. For 8,192 map<int,array<int>> rows with two 128-element values and about 75% missing keys, compaction makes m[1] 29–32% slower, while retained output capacity drops from 4,228,100 to 1,079,300 bytes (review). The kernel numbers miss the downstream effect: buffering operators such as sort reserve memory by capacity, so oversized output can raise memory pressure and spill earlier.

Describe the potential solution

Add a query-level benchmark that feeds nested m[k], element_at and arr[i] results into a buffering operator such as sort, and measures CPU time, peak memory and spills with ample and constrained memory. Use it to tune the shared threshold in nested_compaction.rs, or keep the current one if the results support it.

Additional context

Follow-up to @andygrove's approval of #6845. Both kernels should keep one policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions