Skip to content

Parquet bloom filter pruning drops rows holding the other signed zero: WHERE c = 0.0 misses -0.0 rows (and vice versa) #26491

Open

Description

@hanke580

Describe the bug

A Parquet bloom filter stores hashes of the plain-encoded value, so -0.0 (0x8000000000000000) and +0.0 are
different members. When DataFusion prunes row groups with the bloom filter for c = 0.0 (or c = -0.0,
c IN (…)), it probes only the literal's own bit pattern. A row group whose zeros all have the other sign is answered
"absent" and skipped. SQL = treats -0.0 = 0.0 as true (as does DataFusion's own evaluation without the bloom
filter), so matching rows are silently missing from the result.

bloom_filter_on_read is on by default. So every file with bloom filters on a float column is affected, including
files written by DataFusion itself (bloom_filter_on_write), parquet-java/Spark and parquet-rs. pyarrow and DuckDB return
all matching rows from the same file.

The probe is in datafusion/datasource-parquet/src/bloom_filter.rs (lines 107–108):
ScalarValue::Float64(Some(v)) => sbbf.check(v) and the same for Float32. For a zero literal, both 0.0 and -0.0
should be probed (a row group is kept if either may be present). Alternatively, writers could normalise -0.0 to
+0.0 before hashing, but files written that way already exist.

To Reproduce

import pyarrow as pa, pyarrow.parquet as pq
pq.write_table(pa.table({'id': [1, 2, 3, 4, 5, 6],
                         'c': pa.array([-0.0, -0.0, 0.0, 1.0, -0.0, 2.0], pa.float64())}), 'src.parquet')
SET datafusion.execution.parquet.bloom_filter_on_write = true;
SET datafusion.execution.parquet.max_row_group_size = 2;
CREATE EXTERNAL TABLE s STORED AS PARQUET LOCATION 'src.parquet';
COPY (SELECT * FROM s) TO 'bloom.parquet' STORED AS PARQUET;      -- row groups: (-0.0, -0.0), (0.0, 1.0), (-0.0, 2.0)

SELECT id FROM 'bloom.parquet' WHERE c = 0.0 ORDER BY id;          -- 1, 2, 3        <-- row 5 (-0.0) missing
SELECT id FROM 'bloom.parquet' WHERE c = -0.0 ORDER BY id;         -- 1, 2, 5        <-- row 3 (0.0) missing
SELECT id FROM 'bloom.parquet' WHERE c IN (0.0) ORDER BY id;       -- 1, 2, 3

SET datafusion.execution.parquet.bloom_filter_on_read = false;
SELECT id FROM 'bloom.parquet' WHERE c = 0.0 ORDER BY id;          -- 1, 2, 3, 5     (correct)

pyarrow (pq.read_table('bloom.parquet', filters=[('c', '==', 0.0)])) and DuckDB return 1, 2, 3, 5.

Expected behavior

WHERE c = 0.0 and WHERE c = -0.0 both return rows 1, 2, 3 and 5, with or without bloom filters.

Additional context

  • DataFusion main 290ad15 (2026-10-18), datafusion-cli 55.1.0 built from source; also 91cd7ff (2026-10-15).
  • The min/max path is fine: with bloom_filter_on_read = false all matching rows come back, although the arrow-rs 60
    writer (IEEE_754_TOTAL_ORDER) records the signed zeros as they are (row groups: min = max = -0.0, min = 0.0,
    min = -0.0). Only the bloom-filter probe tells the two zeros apart. A smaller case: COPY of (1, -0.0), (2, 7.0), (3, 0.0), (4, 8.0) with 2-row groups returns only id 3 for c = 0.0 and only id 1 for c = -0.0.

Willingness to contribute

None

Activity

  1. added theissue type on Oct 19, 2026
  2. akshat-lakhera commented on Oct 19, 2026

    @akshat-lakhera

    take

    Plan: probe both zero bit patterns for float literals in the bloom filter pruning, and add a test.

  3. added
    Prolific Authorauthor opened PRs or issues in more than 20 repositories in the last 30 days
    on Oct 24, 2026
  4. akshat-lakhera commented on Oct 31, 2026

    @akshat-lakhera

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Prolific Authorauthor opened PRs or issues in more than 20 repositories in the last 30 daysbugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions