You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cache dictionary value hashes in RepartitionExec #26814
Hash RepartitionExec owns one BatchPartitioner per input partition, but each call to partition_iter invokes create_hashes from scratch. For a dictionary column, that hashes every dictionary value before scattering through the row keys.
Filtering, taking, and slicing commonly produce batches with different keys that retain the same dictionary values Arc:
The O(rows) scatter is required; recomputing the O(dictionary cardinality) value hashes is not. ClickBench Q33 showed the amplification clearly: each producer partition emitted roughly 232 slices sharing about 1.9 million URL values, and summed repartition time increased from 0.50s to 100.66s with dictionary encoding.
BatchPartitioner should cache dictionary value hashes per partitioning expression while the values Arc and repartition hash state are unchanged. A one-entry cache per expression is sufficient: replace it when the pointer changes, so memory cannot grow with the number of batches or dictionaries observed.
Completion criteria:
batches sharing dictionary values hash those values once per input partition
a new values Arc replaces the cached entry
composite hash expressions, null keys/values, slices, and all dictionary key types remain correct
cache allocation is bounded and visible in memory accounting
tests directly count dictionary values hashed
benchmark ClickBench Q10/Q11 and the Q33-shaped grouped/repartitioned path with dictionary reads enabled on both sides
Activity
Rich-T-kid commentedon Oct 27, 2026
take