Skip to content

Avoid aggregate repartitioning for audited date_trunc on single-key timestamp ranges #26448

Closed
benbellick/datafusion
#3

Description

@benbellick

Grouped aggregation can avoid a hash shuffle when transforming a range-partitioned timestamp keeps every group within one input partition.

For example, suppose a table is range-partitioned on timestamp, with partitions split at hourly boundaries:

SELECT
    key,
    date_trunc('hour', timestamp) AS hour,
    SUM(value)
FROM events
GROUP BY key, date_trunc('hour', timestamp);

Each (key, hour) group belongs entirely to one partition, so aggregation can finish independently in each partition. Changing the grouping expression to date_trunc('day', timestamp) would combine rows from multiple hourly partitions and still require a shuffle.

Initially support audited, timezone-less date_trunc expressions over single-key timestamp ranges. Require an explicit, default-deny function capability and validate concrete split boundaries before removing repartitioning. Unsafe or unknown cases must retain the shuffle, and the aggregate-locality proof must not establish join-layout compatibility.

Part of #23569 and the staged work in #25344 and #25345.

Activity

  1. added a commit that references this issue on Oct 23, 2026
    8aad40b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions