Skip to content

Reuse dictionary value hashes across final aggregation input slices #26813

Open

Description

@Rich-T-kid

Related to #26780.

Partial hash aggregation can emit its materialized groups in multiple slices. Slicing a dictionary rewrites the keys but retains the full values array:

partial aggregate dictionary
        |
        +-- slice 1 --+
        +-- slice 2 --+--> FinalAggregate
        +-- slice 3 --+      `-- hashes the same values for every slice

DictionaryGroupValuesColumn already reuses some lookup state while the values Arc is unchanged, but final aggregation's outer group-ID hashing still calls create_hashes for every incoming slice. The work therefore remains O(slices x dictionary cardinality) before the rows are scattered through their keys.

Final aggregation should retain dictionary value hashes across input batches and recompute them only when the values-array identity or hash state changes. The state should live with the final aggregation stream so independent executions do not share mutable caches.

Completion criteria:

  • slices sharing a values Arc hash the dictionary values once
  • replacing the values Arc invalidates the cached hashes
  • composite group keys, null keys/values, and multiple dictionary key types remain correct
  • cache memory is bounded and accounted
  • a PartialAggregate -> Repartition -> FinalAggregate test records the number of dictionary values hashed

Activity

  1. Rich-T-kid commented on Oct 27, 2026

    @Rich-T-kid
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions