Skip to content

Reuse dictionary value hashes in hash joins #26815

Closed as not planned
Closed as not planned

Description

@Rich-T-kid

Related to #26780.

Dictionary join keys can be hashed repeatedly as they pass through partitioning and the join itself:

dictionary input
      |
      +-- RepartitionExec routing hashes
      |
      `-- HashJoinExec build/probe hashes

Within HashJoinExec, consecutive batches also commonly share one dictionary values Arc after scans, filters, or takes. Build and probe hashing currently call create_hashes per batch, so each call hashes the complete values array again before scattering through the keys.

Hash joins should cache dictionary value hashes independently for build and probe streams, keyed by values-array identity and the exact hash state. Repartition hashes cannot be reused unless their seed is compatible; different hash states require separate entries. Equality checks must continue to resolve collisions normally.

Joins between dictionaries with different mappings need explicit coverage: equal logical values can have different keys and dictionary orderings, so cached hashes must always be derived from values rather than raw keys.

Completion criteria:

  • repeated build or probe batches sharing a values Arc hash the values once per compatible hash state
  • changed values arrays and different hash states do not reuse stale hashes
  • tests cover different dictionary mappings, nulls, slices, multiple key types, and forced hash collisions
  • cache memory is bounded and accounted to the join
  • partitioned and collect-left hash-join benchmarks cover low and medium cardinalities

Activity

  1. Rich-T-kid commented on Oct 28, 2026

    @Rich-T-kid
    ContributorAuthor

    Closing as not planned. The final implementation was benchmarked with RLE-to-dictionary enabled on both sides: TPC-H was flat, TPC-DS was 1.32% slower, and ClickBench does not exercise HashJoinExec. The measured benefit does not justify adding this cache to the join path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions