Pruning of floating point Parquet columns is incorrect when NaN is present

Describe the bug

This was mentioned in #15742 (comment) and discussed in detail in apache/parquet-format#221, but datafusion is over-aggressive in pruning floating point columns. The issue appears with predicates of the form x [gt|lt] literal. Consider a column consisting of [1.0, 0.0, -1.0, NaN, -2.0], the max will be 1 and the min -2. A query like select * from ... where x > 2 will return no rows because no chunk exists where max > 2.

To Reproduce

> select * from 'parquet-testing/data/float16_nonzeros_and_nans.parquet' where x > arrow_cast(2.0, 'Float16');
+---+
| x |
+---+
+---+
0 row(s) fetched.

Expected behavior

The above query should return a single row containing NaN.

Additional context

The Parquet community is considering changes to allow for NaN in statistics, with the currently favored approach being adding a new ColumnOrder to the specification. This will correct the issue above, but datafusion will need to check the ColumnOrder to know whether or not floating point statistics can be trusted.

Also note that if/when apache/parquet-format#221 is merged, other predicates such as isnan(x) might be candidates for pruning, but that is an optimization.

@adriangb

cc @adriangb since you've been working in the predicate code recently and may have ideas how to tip off the pruning predicate generation code when floating point pruning is not safe.

I'm not immediately sure. Is the point that the result of max(2.0, NaN) depends on how you define the ordering of floating point numbers wrt NaN, which has two variations?

If so the simplest short term solution would be to not write stats for containers that have NaN. At least results would then be correct.

How do we handle this with nulls 🤔

I'm not immediately sure. Is the point that the result of max(2.0, NaN) depends on how you define the ordering of floating point numbers wrt NaN, which has two variations?

Yes. Different systems treat NaN differently, but IIUC datafusion uses total order for floating point comparison, so (loosely) -NaN < -Inf < -x < -0 < 0 < x < Inf < NaN. This ordering is being proposed for Parquet as well.

If so the simplest short term solution would be to not write stats for containers that have NaN. At least results would then be correct.

Yes, and I believe that's what parquet-java might already do. But many writers do write stats in this case, which leads to the usual backwards compatibility issues. So in my mind the ultimate solution is check ColumnOrder for the predicate column. If it uses the new IEEE_754_TOTAL_ORDER ordering, then proceed as usual. If it uses TYPE_DEFINED_ORDER, then we know NaN may be present but not accounted for, so don't do any pruning. The problem in my head is how to get the ColumnOrder info down into the plan generation code...or do we do extra plan rewriting at the parquet layer? I'm too new with datafusion to have a feel for where the correct place to handle this is.

How do we handle this with nulls 🤔

Different can of worms 😄. I'm not sure what parquet-rs does with a column of NaN mixed with null. Guess I'll go see...

Where is ColumnOrder available? In theory we have access to the full metadata of the parquet file when building the pruning predicate batches / predicate.

Where is ColumnOrder available? In theory we have access to the full metadata of the parquet file when building the pruning predicate batches / predicate.

It's an array in the FileMetaData, so defined once for the entire file.

I've been trying to trace where the plans get built from, and it seems like there are two paths...one from ParquetSource::with_predicate, which seems to happen before the file is opened, and then the path from ParquetOpener::open where we do have access to the file metadata.

Long term I think it will only happen in ParquetOpener::open #15769 precisely for reasons like this 😄

@etseidl

@etseidl I opened #15821 to prove we can pipe the info in. Would you like to continue that PR since you seem to know what the right thing to do with that information is?

@adriangb

Wow, thanks @adriangb! I'll start on it tomorrow!

I can reproduce this with DataFusion engine 55.2.0, the latest stable crate version on 2026-10-28. Here is a four-row PyArrow reproduction, including equality with NaN and the settings needed to recover the rows.

Environment: Linux x86_64, kernel 7.0.0-38-generic, glibc 2.43; CPython 3.14.4; PyArrow 26.0.0. The PyPI package is datafusion==55.0.0; SELECT version() reports Apache DataFusion 55.2.0, x86_64 on linux. The standalone datafusion-cli 55.2.0 gives the same control results below.

Save this as repro_datafusion_pruning.py. In a fresh virtual environment install datafusion==55.0.0 pyarrow==26.0.0, then run python repro_datafusion_pruning.py. The script creates its own input beside itself.

from pathlib import Path
import datafusion as df
import pyarrow as pa
import pyarrow.parquet as pq

path = Path(__file__).with_name('df_nan.parquet')
pq.write_table(pa.table({'x': [float('nan'), 0.0, 0.25, 1.0]}), path,
               row_group_size=3, use_dictionary=False, write_page_index=False)
print('datafusion Python:', df.__version__, 'pyarrow:', pa.__version__)
print(df.SessionContext().sql('SELECT version()').collect())
for pruning, statistics, page_index in [(True, True, True), (False, True, True),
                                      (True, False, True), (True, True, False),
                                      (False, False, False)]:
    config = df.SessionConfig().with_target_partitions(1)
    config.set('datafusion.execution.collect_statistics', str(statistics).lower())
    config.set('datafusion.execution.parquet.pruning', str(pruning).lower())
    config.set('datafusion.execution.parquet.enable_page_index', str(page_index).lower())
    ctx = df.SessionContext(config)
    # register_parquet's explicit pruning option overrides the session setting.
    ctx.register_parquet('t', path, parquet_pruning=pruning)
    print('pruning/statistics/page_index:', pruning, statistics, page_index)
    for query, expected in [
        ('SELECT count(*) AS n FROM t WHERE x >= 0.5', 2),
        ('SELECT count(*) AS n FROM t WHERE x > 0.5', 2),
        ("SELECT count(*) AS n FROM t WHERE x = 'NaN'::DOUBLE", 1),
        ('SELECT max(x) AS m FROM t', 'NaN'),
        ('SELECT max(x+0) AS m FROM t', 'NaN'),
    ]:
        observed = [r for b in ctx.sql(query).collect() for r in b.to_pylist()]
        print(query, 'observed:', observed, 'expected:', expected)

Observed results (both > and >= give the range-count column below):

Row-group pruning	Collect statistics	Page index	range count	NaN equality count	`MAX(x)`
on	on	on	1	0	1.0
off	on	on	2	0	1.0
on	off	on	1	0	NaN
on	on	off	1	0	1.0
off	off	off	2	1	NaN
expected			2	1	NaN

MAX(x+0) returns NaN in every configuration. A separate CLI control with pruning and statistics off but page index on also gives 2, 1, NaN. The file contains no page index, so disabling that feature has no effect here. Disabling row-group pruning alone fixes the range query; equality also needs collected statistics disabled. That distinguishes row-group pruning from an additional use of collected bounds before scanning. The metadata aggregate shortcut is included as supporting evidence for the same unsafe bounds, not as a claim that all consumers share one function.

After the Python script has created df_nan.parquet, the equivalent CLI control is:

datafusion-cli -q --format csv \
  -c "SET datafusion.execution.parquet.pruning = false;" \
  -c "SET datafusion.execution.collect_statistics = false;" \
  -c "SELECT count(*) AS n FROM 'df_nan.parquet' WHERE x >= 0.5;" \
  -c "SELECT count(*) AS n FROM 'df_nan.parquet' WHERE x = 'NaN'::DOUBLE;"
# n = 2, then n = 1; defaults give 1, then 0.

The Python API's register_parquet(..., parquet_pruning=True) default overrides the session pruning option. The reproduction passes the flag explicitly so the off control actually disables pruning. The CLI does not have that registration override.

DataFusion's data type guide identifies Arrow as its execution type system. Arrow Rust's comparison-kernel documentation specifies IEEE totalOrder. DataFusion's own release tests expect positive NaN to compare greater than zero. On the same values in memory, NaN equality matches and the range predicate includes NaN. This example uses only the ordinary positive NaN written by Python, not distinct signs or payloads.

Parquet format 2.14.0 says that when nan_count is absent readers “MUST assume NaNs may be present”. Its TYPE_ORDER reading rules say that when looking for NaN, “min and max should be ignored”. A present count can establish whether NaNs exist; the bounds describe non-NaN values. These reading rules also cover files written before counts were introduced. The generated file has TYPE_ORDER, finite bounds, and no nan_count or page index.

Main runtime status: not tested. I found no packaged official DataFusion nightly; GitHub release APIs offered no executable assets, and a source build was not undertaken. At main 6341b5c9fc30c0215ba326a170eaa294ca1870b7, row-group min/max conversion still supplies the bounds without consulting nan_count. This suggests the pruning path remains relevant, but is a source-based inference, not a main reproduction. PR #24804, addressing #15812, remains open and unmerged as checked on 2026-10-28.

Related: #15812 is the existing report for this root cause. #26490 separately tracks replacing a float column with a constant from min == max; this reproduction keeps distinct finite bounds in the NaN-containing group to avoid that confound. #24432 is the separate grouped-accumulator bug, already fixed on main; the MAX queries above have no GROUP BY.

Navigation Menu

Repository navigation

Pruning of floating point Parquet columns is incorrect when `NaN` is present #15812

Describe the bug

To Reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Pruning of floating point Parquet columns is incorrect when NaN is present #15812

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Activity

etseidl commented on Apr 22, 2025

adriangb commented on Apr 22, 2025

adriangb commented on Apr 22, 2025

etseidl commented on Apr 22, 2025

adriangb commented on Apr 22, 2025

etseidl commented on Apr 22, 2025

adriangb commented on Apr 22, 2025

adriangb commented on Apr 23, 2025

etseidl commented on Apr 23, 2025

chicagobuss commented on Oct 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Pruning of floating point Parquet columns is incorrect when `NaN` is present #15812