Repository navigation
Support file stats on nested fields #6389
Description
Discussed in #5917
Problem
Currently, only top-level field stats are stored in the vortex file footer:
While chunk-level pruning on nested fields already works, file-level stats are missing, preventing whole-file pruning on nested fields or other optimizations. For example, we have some queries that can 100% be satisfied just by nested file stats, removing extra object storage reads.
Proposed approach
Store stats via post-order DType walk. Each leaf and nullable struct gets a stats set entry:
{name=utf8?, age=i32} → [name stats_set, age stats_set]
{name=utf8?, age=i32}? → [name stats_set, age stats_set, struct stats_set (null count)]
Extend the stats set flatbuffer with an is_nested: bool = false field for backward compatibility. When false, existing behavior is preserved; when true, entries follow the post-order walk.
Lists: For list(T), store stats only for the element column.
Reacted by Joe Isaacs
Activity
asubiotto commentedon Feb 10, 2026
cc @joseph-isaacs
joseph-isaacs commentedon Sep 1, 2026
We should unify the zoned layout and the footer stats. The zoned layout uses AggRef Partials and the footer stats use a closed enum.
We should use partials everywhere
thorfour commentedon Sep 2, 2026
@joseph-isaacs Can I be assigned this issue? I would like to implement this behavior
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (#6389)
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (vortex-data#6389)
Support file stats on nested struct fields (vortex-data#6389)
Statistics for nested columns (#9775)
thorfour commentedon Nov 5, 2026
github.com//pull/9775 Implements this.