Skip to content

Consolidate date/timestamp truncation and xxhash64 with upstream DataFusion implementations #5103

Open

Description

@andygrove

What is the problem the feature request solves?

Beyond arrow kernels, two areas duplicate functionality now available upstream in DataFusion / datafusion-spark (54.1):

  1. Date/timestamp truncation: native/spark-expr/src/kernels/temporal.rs implements trunc(date, fmt) and date_trunc(fmt, ts) by hand. Arrow has no truncation kernel, but datafusion_functions::datetime::date_trunc is timezone-aware (chrono-tz), supports all the granularities Spark needs with Monday-based weeks, and preserves the input tz annotation. Upstream datafusion-spark's trunc already composes it with the same alias map (yy/yyyy to year, mm/mon to month) via a cast sandwich.
  2. xxhash64: datafusion-spark 54.1 ships a Spark-compatible SparkXxhash64 (twox-hash based, with its own type-dispatch machinery). Comet's hash_funcs/xxhash64.rs could potentially be rewired to it. Note datafusion-spark has no murmur3, so Comet's murmur3.rs stays.

Describe the potential solution

  • Truncation: wire date_trunc at the physical level for the scalar-format paths (datafusion-spark's trunc/date_diff are simplify-only UDFs whose invoke_with_args returns an internal error, so they cannot be called from Comet's physical-plan pipeline directly). Keep custom code for: per-row-format variants (DataFusion requires scalar granularity), dictionary inputs, and the Spark session-timezone pre-shift.
  • Requires differential DST-boundary testing first: Comet resolves ambiguous local times to the earliest occurrence and falls back to the original instant on gaps (test_timestamp_trunc_dst_boundary); DataFusion's fine-versus-coarse granularity handling needs case-by-case verification.
  • xxhash64: verify type coverage before rewiring (Comet's macro additionally handles the small/large Decimal128 split, Map, FixedSizeList, and Struct fast paths).

Additional context

Related in spirit: string_funcs/contains.rs and array_funcs/array_slice.rs each document an upstream DataFusion issue that, once fixed, lets the Comet file be dropped entirely.

Found during an audit of native code that replicates existing arrow-rs kernels.

Activity

  1. sam-1112 commented on Sep 13, 2026

    @sam-1112
    Contributor

    take

  2. sam-1112 commented on Sep 13, 2026

    @sam-1112
    Contributor

    I’ll split #5103 into two focused PRs:

    • Truncation: migrate compatible scalar-format trunc(date, fmt) and date_trunc(fmt, timestamp) paths, with Spark compatibility tests. Keep Comet’s per-row-format, dictionary, and fallback paths unchanged, and verify timestamp behavior across granularities and DST boundaries as part of the migration. (feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956)
    • xxhash64: add differential coverage for seeds, nulls, decimals, nested types, and other relevant edge cases, then route only the behaviors established as compatible through the DataFusion Spark implementation. Unsupported or incompatible cases will continue using the existing Comet path.(feat: route compatible xxhash64 args through SparkXxhash64 #5960 )

    I’ll keep these as separate PRs so the truncation and hash changes can be reviewed independently.

  3. added a commit that references this issue on Sep 18, 2026
    bae7f16
  4. sam-1112 commented on Oct 6, 2026

    @sam-1112
    Contributor

    With #5956 and #5960 merged, I believe the consolidation work tracked by this issue is complete.

    The remaining follow-up work is tracked separately:

    I think this issue can now be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions