You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Reuse dictionary value hashes across final aggregation input slices #26813
Partial hash aggregation can emit its materialized groups in multiple slices. Slicing a dictionary rewrites the keys but retains the full values array:
partial aggregate dictionary
|
+-- slice 1 --+
+-- slice 2 --+--> FinalAggregate
+-- slice 3 --+ `-- hashes the same values for every slice
DictionaryGroupValuesColumn already reuses some lookup state while the values Arc is unchanged, but final aggregation's outer group-ID hashing still calls create_hashes for every incoming slice. The work therefore remains O(slices x dictionary cardinality) before the rows are scattered through their keys.
Final aggregation should retain dictionary value hashes across input batches and recompute them only when the values-array identity or hash state changes. The state should live with the final aggregation stream so independent executions do not share mutable caches.
Completion criteria:
slices sharing a values Arc hash the dictionary values once
replacing the values Arc invalidates the cached hashes
composite group keys, null keys/values, and multiple dictionary key types remain correct
cache memory is bounded and accounted
a PartialAggregate -> Repartition -> FinalAggregate test records the number of dictionary values hashed
Activity
Rich-T-kid commentedon Oct 27, 2026
take