You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Consolidate date/timestamp truncation and xxhash64 with upstream DataFusion implementations #5103
Beyond arrow kernels, two areas duplicate functionality now available upstream in DataFusion / datafusion-spark (54.1):
Date/timestamp truncation: native/spark-expr/src/kernels/temporal.rs implements trunc(date, fmt) and date_trunc(fmt, ts) by hand. Arrow has no truncation kernel, but datafusion_functions::datetime::date_trunc is timezone-aware (chrono-tz), supports all the granularities Spark needs with Monday-based weeks, and preserves the input tz annotation. Upstream datafusion-spark's trunc already composes it with the same alias map (yy/yyyy to year, mm/mon to month) via a cast sandwich.
xxhash64: datafusion-spark 54.1 ships a Spark-compatible SparkXxhash64 (twox-hash based, with its own type-dispatch machinery). Comet's hash_funcs/xxhash64.rs could potentially be rewired to it. Note datafusion-spark has no murmur3, so Comet's murmur3.rs stays.
Describe the potential solution
Truncation: wire date_trunc at the physical level for the scalar-format paths (datafusion-spark's trunc/date_diff are simplify-only UDFs whose invoke_with_args returns an internal error, so they cannot be called from Comet's physical-plan pipeline directly). Keep custom code for: per-row-format variants (DataFusion requires scalar granularity), dictionary inputs, and the Spark session-timezone pre-shift.
Requires differential DST-boundary testing first: Comet resolves ambiguous local times to the earliest occurrence and falls back to the original instant on gaps (test_timestamp_trunc_dst_boundary); DataFusion's fine-versus-coarse granularity handling needs case-by-case verification.
xxhash64: verify type coverage before rewiring (Comet's macro additionally handles the small/large Decimal128 split, Map, FixedSizeList, and Struct fast paths).
Additional context
Related in spirit: string_funcs/contains.rs and array_funcs/array_slice.rs each document an upstream DataFusion issue that, once fixed, lets the Comet file be dropped entirely.
Found during an audit of native code that replicates existing arrow-rs kernels.
Truncation: migrate compatible scalar-format trunc(date, fmt) and date_trunc(fmt, timestamp) paths, with Spark compatibility tests. Keep Comet’s per-row-format, dictionary, and fallback paths unchanged, and verify timestamp behavior across granularities and DST boundaries as part of the migration. (feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956)
xxhash64: add differential coverage for seeds, nulls, decimals, nested types, and other relevant edge cases, then route only the behaviors established as compatible through the DataFusion Spark implementation. Unsupported or incompatible cases will continue using the existing Comet path.(feat: route compatible xxhash64 args through SparkXxhash64 #5960 )
I’ll keep these as separate PRs so the truncation and hash changes can be reviewed independently.
Activity
sam-1112 commentedon Sep 13, 2026
take
sam-1112 commentedon Sep 13, 2026
I’ll split #5103 into two focused PRs:
trunc(date, fmt)anddate_trunc(fmt, timestamp)paths, with Spark compatibility tests. Keep Comet’s per-row-format, dictionary, and fallback paths unchanged, and verify timestamp behavior across granularities and DST boundaries as part of the migration. (feat: Use DataFusion date_trunc for scalar-format timestamp truncation #5956)I’ll keep these as separate PRs so the truncation and hash changes can be reviewed independently.
feat: route compatible xxhash64 args through SparkXxhash64 (apache#5960)
sam-1112 commentedon Oct 6, 2026
With #5956 and #5960 merged, I believe the consolidation work tracked by this issue is complete.
The remaining follow-up work is tracked separately:
I think this issue can now be closed.