Repository navigation
Merging Statistics is slow when sum statistic is present #15809
Description
Is your feature request related to a problem or challenge?
If you have a datasource that reports a sum statistic then merging statistics for a file group is considerably slower when compared to cases without sum statistic. Profiling reveals it's mostly due to Precision<ScalarValue>::add doing a lot of extra work.
Describe the solution you'd like
Ideally we'd transpose the stats aggregation and aggregate them column wise instead of column group wise
Describe alternatives you've considered
No response
Additional context
Screenshots of Parquet and Vortex profiles from clickbench Q0 which is all just getting statistics. You can notice that roughly 50% of Vortex time is spent in aggregating stats
Activity
[-]Merging Statistics is slow[/-][+]Merging Statistics is slow when sum statistic is present[/+]Stop reporting sum stats to datafusion (#3088)
adriangb commentedon Oct 15, 2025
Discussed today in a call: could we refactor the logic to combine statistics by first converting them to a RecordBatch and then evaluating the combination in a vectorized fashion
feat: Reduce allocations for aggregating `Statistics` (#20768)
feat: Reduce allocations for aggregating `Statistics` (apache#20768)