Skip to content

Cache dictionary value hashes in RepartitionExec #26814

Open

Description

@Rich-T-kid

Related to #26780.

Hash RepartitionExec owns one BatchPartitioner per input partition, but each call to partition_iter invokes create_hashes from scratch. For a dictionary column, that hashes every dictionary value before scattering through the row keys.

Filtering, taking, and slicing commonly produce batches with different keys that retain the same dictionary values Arc:

shared dictionary values
        |
        +-- batch 1 keys -> hash all values -> scatter rows
        +-- batch 2 keys -> hash all values -> scatter rows
        +-- batch 3 keys -> hash all values -> scatter rows

The O(rows) scatter is required; recomputing the O(dictionary cardinality) value hashes is not. ClickBench Q33 showed the amplification clearly: each producer partition emitted roughly 232 slices sharing about 1.9 million URL values, and summed repartition time increased from 0.50s to 100.66s with dictionary encoding.

BatchPartitioner should cache dictionary value hashes per partitioning expression while the values Arc and repartition hash state are unchanged. A one-entry cache per expression is sufficient: replace it when the pointer changes, so memory cannot grow with the number of batches or dictionaries observed.

Completion criteria:

  • batches sharing dictionary values hash those values once per input partition
  • a new values Arc replaces the cached entry
  • composite hash expressions, null keys/values, slices, and all dictionary key types remain correct
  • cache allocation is bounded and visible in memory accounting
  • tests directly count dictionary values hashed
  • benchmark ClickBench Q10/Q11 and the Q33-shaped grouped/repartitioned path with dictionary reads enabled on both sides

Activity

  1. Rich-T-kid commented on Oct 27, 2026

    @Rich-T-kid
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions