Repository navigation
Missing Parquet row-group statistics can produce incorrect file MIN/MAX #25226
Description
Describe the bug
When a nonempty row group lacks min/max statistics, DataFusion can still report another group's bounds as exact for the entire file. MIN/MAX queries can then return incorrect results.
To Reproduce
Generate the file with Rust (parquet 59.3, re-exported by datafusion, and tempfile). Enable statistics for the first row group and disable them for the second using the public writer APIs. Both groups contain non-null strings. This explicitly creates missing statistics; long strings alone do not cause parquet-rs to omit bounds.
use std::{fs::File, sync::Arc};
fn main() {
use datafusion::parquet::column::writer::ColumnWriterImpl;
use datafusion::parquet::data_type::{ByteArray, ByteArrayType};
use datafusion::parquet::file::properties::{EnabledStatistics, WriterProperties};
use datafusion::parquet::file::writer::{
SerializedFileWriter, SerializedPageWriter, TrackedWrite,
};
use datafusion::parquet::schema::parser::parse_message_type;
let dir = tempfile::tempdir().unwrap();
let path = "/tmp/parquet_missing_bounds.parquet";
let column_path = dir.path().join("column.pages");
let schema = Arc::new(
parse_message_type("message schema { REQUIRED BINARY a (UTF8); }").unwrap(),
);
let mut writer = SerializedFileWriter::new(
File::create(path).unwrap(),
schema,
Arc::new(WriterProperties::default()),
)
.unwrap();
let long_value = "z".repeat(8192);
for (values, statistics) in [
(["a", "b"], EnabledStatistics::Chunk),
(
[long_value.as_str(), long_value.as_str()],
EnabledStatistics::None,
),
] {
let properties = Arc::new(
WriterProperties::builder()
.set_statistics_enabled(statistics)
.build(),
);
let mut buffer = TrackedWrite::new(File::create(&column_path).unwrap());
let mut column = ColumnWriterImpl::<ByteArrayType>::new(
writer.schema_descr().column(0),
properties,
Box::new(SerializedPageWriter::new(&mut buffer)),
);
let values = values.map(ByteArray::from);
column.write_batch(&values, None, None).unwrap();
let result = column.close().unwrap();
assert_eq!(
result.metadata.statistics().is_some(),
statistics == EnabledStatistics::Chunk
);
drop(buffer);
let mut group = writer.next_row_group().unwrap();
group
.append_column(&File::open(&column_path).unwrap(), result)
.unwrap();
group.close().unwrap();
}
writer.close().unwrap();
}Query the generated file in DataFusion:
CREATE EXTERNAL TABLE t STORED AS PARQUET
LOCATION '/tmp/parquet_missing_bounds.parquet';
SELECT LENGTH(MIN(a)), LENGTH(MAX(a)) FROM t;Before the fix: 1, 1. Expected: 1, 8192. The maximum is in the second row group, whose missing bound must not be ignored.
Expected behavior
A missing bound in a potentially nonempty, non-null row group must prevent reporting that file bound as exact.
Additional context
find this when work on #25220
Activity
haohuaijin commentedon Sep 12, 2026
take
fix: discard parquet bounds when row group statistics are missing (ap…
fix: discard parquet bounds when row group statistics are missing (ap…