Repository navigation
Description
Describe the bug
A Parquet bloom filter stores hashes of the plain-encoded value, so -0.0 (0x8000000000000000) and +0.0 are
different members. When DataFusion prunes row groups with the bloom filter for c = 0.0 (or c = -0.0,
c IN (…)), it probes only the literal's own bit pattern. A row group whose zeros all have the other sign is answered
"absent" and skipped. SQL = treats -0.0 = 0.0 as true (as does DataFusion's own evaluation without the bloom
filter), so matching rows are silently missing from the result.
bloom_filter_on_read is on by default. So every file with bloom filters on a float column is affected, including
files written by DataFusion itself (bloom_filter_on_write), parquet-java/Spark and parquet-rs. pyarrow and DuckDB return
all matching rows from the same file.
The probe is in datafusion/datasource-parquet/src/bloom_filter.rs (lines 107–108):
ScalarValue::Float64(Some(v)) => sbbf.check(v) and the same for Float32. For a zero literal, both 0.0 and -0.0
should be probed (a row group is kept if either may be present). Alternatively, writers could normalise -0.0 to
+0.0 before hashing, but files written that way already exist.
To Reproduce
import pyarrow as pa, pyarrow.parquet as pq
pq.write_table(pa.table({'id': [1, 2, 3, 4, 5, 6],
'c': pa.array([-0.0, -0.0, 0.0, 1.0, -0.0, 2.0], pa.float64())}), 'src.parquet')SET datafusion.execution.parquet.bloom_filter_on_write = true;
SET datafusion.execution.parquet.max_row_group_size = 2;
CREATE EXTERNAL TABLE s STORED AS PARQUET LOCATION 'src.parquet';
COPY (SELECT * FROM s) TO 'bloom.parquet' STORED AS PARQUET; -- row groups: (-0.0, -0.0), (0.0, 1.0), (-0.0, 2.0)
SELECT id FROM 'bloom.parquet' WHERE c = 0.0 ORDER BY id; -- 1, 2, 3 <-- row 5 (-0.0) missing
SELECT id FROM 'bloom.parquet' WHERE c = -0.0 ORDER BY id; -- 1, 2, 5 <-- row 3 (0.0) missing
SELECT id FROM 'bloom.parquet' WHERE c IN (0.0) ORDER BY id; -- 1, 2, 3
SET datafusion.execution.parquet.bloom_filter_on_read = false;
SELECT id FROM 'bloom.parquet' WHERE c = 0.0 ORDER BY id; -- 1, 2, 3, 5 (correct)pyarrow (pq.read_table('bloom.parquet', filters=[('c', '==', 0.0)])) and DuckDB return 1, 2, 3, 5.
Expected behavior
WHERE c = 0.0 and WHERE c = -0.0 both return rows 1, 2, 3 and 5, with or without bloom filters.
Additional context
- DataFusion main 290ad15 (2026-10-18),
datafusion-cli 55.1.0built from source; also 91cd7ff (2026-10-15). - The min/max path is fine: with
bloom_filter_on_read = falseall matching rows come back, although the arrow-rs 60
writer (IEEE_754_TOTAL_ORDER) records the signed zeros as they are (row groups:min = max = -0.0,min = 0.0,
min = -0.0). Only the bloom-filter probe tells the two zeros apart. A smaller case:COPYof(1, -0.0), (2, 7.0), (3, 0.0), (4, 8.0)with 2-row groups returns only id 3 forc = 0.0and only id 1 forc = -0.0.
Willingness to contribute
None
Activity
akshat-lakhera commentedon Oct 19, 2026
take
Plan: probe both zero bit patterns for float literals in the bloom filter pruning, and add a test.
akshat-lakhera commentedon Oct 31, 2026
take