Repository navigation
Integrate SumV2 into stats #9595
Description
Tracking Issue: #9591
Statistics migration: #10177
Integrate SumV2 into statistics and file summaries without summing finalized chunk results as though they were raw input values.
- Merge partial states using the original input and result dtypes, retaining overflow and the distinction between empty, all-null, and valid results.
- Cover decimal precision across chunk boundaries, including File sums can widen decimal precision twice when combining chunks #10581.
- Define NaN handling and floating-point chunk grouping explicitly. Include NaNs produced by combining infinities, rather than only NaNs present in the input.
- Preserve the distinction between legacy
SumandSumV2when reading historical statistics and choosing cache keys. - Check the returned footer and the reopened footer, including any chunk-cache metadata used by compression.
Merged #10328 fixes lost integer overflow in the legacy file sum path by omitting the result when a chunk sum is null. The broader accumulator-state approach in closed #10228 was not merged. Its regression cases remain useful when implementing this migration.
Activity
connortsui20 commentedon Oct 27, 2026
(generated by codex)
Signed overflow also depends on batch and encoding boundaries at
088118bb3f.Default
Sumreturns null for primitive[i64::MAX - 5, 10, -10], but returnsi64::MAX - 5when split into[i64::MAX - 5]and[10, -10], represented as those chunks, or represented as Sparse.SumV2andSumwithskip_nans=falseagree across simple primitive splits. However, after accumulating prefix[i64::MAX - 5], a RunEnd, Sparse or Chunked suffix representing[10, -10]returnsi64::MAX - 5, while the decoded suffix returns null.Those kernels return a from-zero partial of zero, which loses the overflowing intermediate prefix. Retaining an overflow flag alone does not resolve this. #7843 already identifies signed partial aggregation as unsupported under strict Vortex semantics. The statistics migration needs to preserve that restriction or define compatible overflow semantics.
The existing Sparse test helper also shares cached statistics between its two paths. Comparing against a separately decoded array exposes the failure.