[EPIC] Handle -0.0 (and NaN) float keys in one place instead of per call site

In short: this keeps the approach from #22835, which maps -0.0 to 0.0 before Arrow compares anything, and completes it in three ways:

apply the same mapping to sort order, not only to equality
route it through one module instead of individual call sites
add a differential test that finds the call sites that still skip it

Is your feature request related to a problem or challenge?

DataFusion treats -0.0 and 0.0 as equal (#22835), like PostgreSQL, DuckDB and Spark. Arrow's kernels don't: they compare floats by bits and sort them by IEEE total order, where -0.0 < 0.0. #22835 bridged the gap by rewriting -0.0 to 0.0 before some Arrow calls. Every comparison, sort or hash that skips the rewrite still treats them as different, and these keep turning up one at a time: #25186 (IN list), #25766 (window PARTITION BY), #26115 (sort-merge join), #24375 (ASOF). #26091 asks for the same treatment for NaN.

The bigger problem is that only equality was fixed, not ordering. Sorting still puts -0.0 before 0.0. For one column that is harmless. When another value follows the zero (multi-column, list or struct keys), sort order and equality disagree, and anything that sorts and then compares gets wrong results:

SELECT a, b FROM (VALUES (arrow_cast(-0.0, 'Float64'), 1), (0.0, 0)) t(a, b) ORDER BY a, b;
-- got (-0.0, 1), (0.0, 0)
-- expected (0.0, 0), (-0.0, 1): `a` ties, so `b` decides

SELECT a, rank() OVER (ORDER BY a) FROM (VALUES (arrow_cast(-0.0, 'Float64')), (0.0)) t(a);
-- got ranks 1, 2; expected 1, 1

#26115 is the same problem in a sort-merge join, which drops matching rows. More cases that are still wrong are listed at the end.

Describe the solution you'd like

Map every float to one representative before comparing, sorting or hashing it: -0.0 → 0.0, and, if #26091 is accepted, every NaN → one NaN. On mapped values, Arrow's bit equality and total order already match SQL semantics, for nested types too. So no SQL-specific kernels are needed; the mapping just has to be applied everywhere.

One new module owns this, for example datafusion_common::float_semantics (the name Comet uses for the same idea):
- the mapping
- wrappers around the Arrow kernels (eq, sort, RowConverter, make_comparator, ...)
- SQL-order comparison for code that compares by hand (compare_rows, evaluate_partition_ranges, IN list, array_*)
Operators map their keys once, when they evaluate them. For flat comparisons the module can compare in SQL order directly instead of copying, as Compare floats in Spark's order without normalizing copies of the operands datafusion-comet#6590 does.
One differential test. It runs special floats (±0, NaN variants, nested values) through operators that must agree: hash vs sort-merge joins, hash vs sorted aggregation, window partitions vs GROUP BY, ORDER BY vs a reference sort. This catches the next case whatever code path it comes from.
Optionally, a lint: list the raw Arrow kernels in disallowed-methods in clippy.toml, so new code goes through the module. A lint can't see hand-written comparisons, so the test is the main guard.

Values don't change: -0.0 still prints as -0.0. ScalarValue equality stays bitwise, because it is expression identity (1 / -0.0 ≠ 1 / 0.0). The visible change is that ORDER BY a treats -0.0 and 0.0 as ties, as PostgreSQL does.

Steps:

Module, differential test and the sort path (closes SortMergeJoin misses rows for nested float keys containing -0.0 #26115)
Move the feat: Support IEEE 754 negative zero semantics #22835 call sites onto the module (refactor: Centralize float normalization #26595)
Window partitions, rank and RANGE frames (closes Window PARTITION BY puts 0.0 and -0.0 in separate partitions #25766)
IN list, CASE, nullif, array_*, min/max on nested values, count(DISTINCT)
NaN, if feat: Treat all NaNs as the same value (-NaN = NaN) #26091 is accepted
Optional lint

Describe alternatives you've considered

Keep fixing call sites one by one. Fixes have arrived one at a time since feat: Support IEEE 754 negative zero semantics #22835 and the list keeps growing.
Wrap key expressions in the plan, like Spark's NormalizeFloatingNumbers. fix: treat signed zeros as equal in window PARTITION BY #26020 and feat: support floating-point ASOF equality keys #24375 each add a NormalizeFloatZeroExpr this way. Every float sort key would need the wrapper too, which changes how the optimizer matches orderings and partitioning. It also misses expression-level equality (IN, CASE, array_*). It would be good to agree on a direction before those two PRs merge.

Additional context

datafusion-comet hit the same problem with Spark semantics and is fixing it the same way: [EPIC] Match Spark's -0.0 and NaN semantics systematically instead of per expression datafusion-comet#6385.
Arrow uses total order on purpose (Add sql-compliant feature for enabling sql-compliant kernel behavior arrow-rs#2568, Make consistent behavior on zeros equality on floating point types arrow-rs#3509), so this belongs in DataFusion.
Cost: nothing for non-float columns. For float keys, one pass per key column per batch, which is small next to a sort or join. Flat comparisons should compare in SQL order directly; Comet measured copying at about +50% there.

Other cases still wrong on main (checked Oct 29)

z is arrow_cast(-0.0, 'Float64').

Query	Got	Expected
`ORDER BY a, b LIMIT 1` over `(z, 1), (0.0, 0)`	`(-0.0, 1)`	`(0.0, 0)`
`sum(b) OVER (ORDER BY a RANGE BETWEEN CURRENT ROW AND CURRENT ROW)` over `(z, 1), (0.0, 2)`	1, 2	3, 3
`row_number() OVER (PARTITION BY a)` over `z, 0.0`	1, 1	1, 2
`first_value(b ORDER BY a, b)` over `(z, 1), (0.0, 0)`	1	0
`DISTINCT ON (g) g, b ... ORDER BY g, a, b` over `(1, z, 1), (1, 0.0, 0)`	b = 1	b = 0
`ORDER BY k` over `k = [z, 1.0]`, `[0.0, 0.0]`	`[-0.0, 1.0]` first	`[0.0, 0.0]` first
`min(k)`, `max(k)` over the same lists	`[-0.0, 1.0]`, `[0.0, 0.0]`	`[0.0, 0.0]`, `[-0.0, 1.0]`
`z IN (0.0, 1.0)`, Float64 and Float32	false	true
`z IN (0.0, 1.0, ..., 25.0)` (26 values)	false	true
`array_has(make_array(z), 0.0)`, `array_has_any`, `array_has_all`	false	true
`array_position(make_array(z, 1.0), 0.0)`	NULL	1
`array_replace(make_array(z, 1.0), 0.0, 9.0)`	`[-0.0, 1.0]`	`[9.0, 1.0]`
`array_remove(make_array(z, 1.0), 0.0)`	`[-0.0, 1.0]`	`[1.0]`
`nullif(z, 0.0)`	-0.0	NULL
`CASE z WHEN 0.0 THEN 'eq' ELSE 'ne' END`	ne	eq
`count(DISTINCT a), count(DISTINCT b)` over `(z, 1), (0.0, 0)`	2, 2	1, 2
`count(DISTINCT a) OVER ()` over `z, 0.0`	2	1

I think someone proposed this idea before, but I can't remember where.

@jayzhan211

@jayzhan211 This looks great. I am happy to help. I filed #26091 a few days ago which is part of this epic.

take

refactor: centralize float canonicalization in `datafusion_common::fl…

This is great if it can be done without (too much) performance impact. I was struggling coming up with a clean implementation of #25186 .

This is great if it can be done without (too much) performance impact. I was struggling coming up with a clean implementation of #25186 .

Agree +1

refactor: Centralize float normalization (apache#26595)

@jayzhan211

@jayzhan211 Do you have some input regarding the differential test ? Would be slt sufficient here or do you picture something more clever ?

@mkleen

@mkleen slt is right for the cases we already know: each fix PR should pin its rows from the table in the description in slt. The differential test is for the cases nobody has found yet, so it can't rely on hand-written expected output.

The rule: changing a -0.0 to 0.0 must not change any result, apart from how -0.0 prints. So the expected result comes from DataFusion itself, run on a copy where every -0.0 is 0.0.

Generate tables whose float columns (flat, list, struct) draw from a small pool (0.0, 1.0, -1.0, inf, one NaN), so keys tie often. Add a unique id.
Make a second copy by flipping the sign of some zeros, both in the data and in the zero literals of the query, since constant folding is a separate path (today 0.0 IN (-0.0) is false while -0.0 = 0.0 is true).
Run the query on both copies under the same settings, print -0.0 as 0.0, and compare. Repeat under several settings: hash vs sort-merge join, batch size 1, 1 vs 4 partitions, a memory limit that forces spilling, and both copies read from Parquet with small row groups.

This also covers code with only one implementation (IN, CASE, nullif, array_*), where comparing hash join with sort-merge join can't help. It would replace the "operators that must agree" test in the description; I'll update the description if you agree.

I tried a quick SQL version on main: 25 queries over 4 rows, queries below. It flags every case in the table, plus some new ones, which I've added to the table:

count(DISTINCT a) counts -0.0 and 0.0 separately when there is a second DISTINCT aggregate, or when it is a window function. On its own it passes, because it is planned as a GROUP BY.
array_remove, and array_has(k, 0.0) on a list column. With a column a, array_has(make_array(a), 0.0) passes, but only because it is rewritten to a = 0.0 and never reaches the array_has kernel.
Float64 z IN (0.0, 1.0) now returns false too, not only Float32.
With prefer_hash_join = false, a join on a List(Float64) key returns 3 of 6 pairs, and two of the dropped pairs contain no -0.0 (SortMergeJoin misses rows for nested float keys containing -0.0 #26115).

GROUP BY and the flat equi-join pass.

For the real test I'd suggest Rust in datafusion/core/tests/fuzz_cases/. aggregation_fuzzer shows how to run a query under random settings, but its check sorts the rows, which would hide the ORDER BY bugs here. Some rules:

A fixed list of query templates, not random SQL: ORDER BY (several columns, list keys, with LIMIT), window PARTITION BY / rank / RANGE frames, GROUP BY / DISTINCT / DISTINCT ON, count(DISTINCT), equi and range joins, set operations, IN, CASE, nullif, array_*, min/max on lists.
Each query must have one correct answer. An ORDER BY that picks or numbers rows (LIMIT, first_value, DISTINCT ON, row_number, lag, ROWS frames) ends with id. rank and RANGE frames don't, because their ties are what we test. Compare ORDER BY output in order and everything else as a multiset.
No arithmetic on the float columns (-x, x * y), since it can create -0.0 in the zero-only copy. No functions that can see the sign (1 / x, atan2, cot, casts to string).
A list of (query, setting) pairs that are still wrong, each with its issue. Random data can only check that unlisted pairs pass. A fixed dataset where each listed pair is known to fail also checks that listed pairs still fail, so each fix PR has to remove its entries.
Once feat: Treat all NaNs as the same value (-NaN = NaN) #26091 is decided, add NaN payloads to the pool and to step 2.

Does this match what you had in mind? If so, this test could be the next PR, before the sort-order change, and I'm happy to review it.

Queries (each runs on raw and on canon in place of {t})

CREATE TABLE raw AS SELECT
  column1 AS id, column2 AS a, arrow_cast(column2, 'Float32') AS a32,
  column3 AS b, 1 AS g, column4 AS k
FROM (VALUES
  (1, arrow_cast(-0.0, 'Float64'), 1, make_array(arrow_cast(-0.0, 'Float64'), 1.0)),
  (2, arrow_cast(0.0, 'Float64'), 0, make_array(0.0, 0.0)),
  (3, arrow_cast(1.0, 'Float64'), 2, make_array(1.0)),
  (4, arrow_cast(-0.0, 'Float64'), 3, make_array(0.0, 1.0))
);
CREATE TABLE canon AS SELECT
  column1 AS id, column2 AS a, arrow_cast(column2, 'Float32') AS a32,
  column3 AS b, 1 AS g, column4 AS k
FROM (VALUES
  (1, arrow_cast(0.0, 'Float64'), 1, make_array(0.0, 1.0)),
  (2, arrow_cast(0.0, 'Float64'), 0, make_array(0.0, 0.0)),
  (3, arrow_cast(1.0, 'Float64'), 2, make_array(1.0)),
  (4, arrow_cast(0.0, 'Float64'), 3, make_array(0.0, 1.0))
);
SELECT a, b FROM {t} WHERE id <= 2 ORDER BY a, b;
SELECT id, rank() OVER (ORDER BY a) FROM {t} ORDER BY id;
SELECT id, sum(b) OVER (ORDER BY a RANGE BETWEEN CURRENT ROW AND CURRENT ROW) FROM {t} ORDER BY id;
SELECT max(rn) FROM (SELECT row_number() OVER (PARTITION BY a) rn FROM {t} WHERE id <= 2);
SELECT first_value(b ORDER BY a, b) FROM {t} WHERE id <= 2;
SELECT DISTINCT ON (g) g, b FROM {t} ORDER BY g, a, b;
SELECT id FROM {t} WHERE id <= 2 ORDER BY k;
SELECT min(k), max(k) FROM {t} WHERE id <= 2;
SELECT id, a32 IN (0.0, 1.0) FROM {t} ORDER BY id;
SELECT id, a IN (0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,22.0,23.0,24.0,25.0) FROM {t} ORDER BY id;
SELECT id, array_has(make_array(a), 0.0), array_has_any(make_array(a), make_array(0.0)), array_has_all(make_array(a), make_array(0.0)) FROM {t} ORDER BY id;
SELECT id, array_position(k, 0.0) FROM {t} ORDER BY id;
SELECT id, array_replace(k, 0.0, 9.0) FROM {t} ORDER BY id;
SELECT id, nullif(a, 0.0) FROM {t} ORDER BY id;
SELECT id, CASE a WHEN 0.0 THEN 'eq' ELSE 'ne' END FROM {t} ORDER BY id;
SELECT count(*) c FROM {t} GROUP BY a ORDER BY c;
SELECT count(*) FROM {t} x JOIN {t} y ON x.a = y.a;
SELECT count(DISTINCT a) FROM {t};
SELECT count(*) FROM {t} x JOIN {t} y ON x.k = y.k;
SELECT a, b FROM {t} WHERE id <= 2 ORDER BY a, b LIMIT 1;
SELECT id, array_has(k, 0.0) FROM {t} ORDER BY id;
SELECT id, a IN (0.0, 1.0) FROM {t} ORDER BY id;
SELECT count(DISTINCT a), count(DISTINCT b) FROM {t};
SELECT DISTINCT count(DISTINCT a) OVER () FROM {t};
SELECT id, array_remove(k, 0.0) FROM {t} ORDER BY id;

@jayzhan211

@jayzhan211 Thanks a lot for the input!

Navigation Menu

Repository navigation

[EPIC] Handle `-0.0` (and NaN) float keys in one place instead of per call site #26138

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[EPIC] Handle -0.0 (and NaN) float keys in one place instead of per call site #26138

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

jayzhan211 commented on Oct 8, 2026

mkleen commented on Oct 8, 2026

mkleen commented on Oct 22, 2026

geoffreyclaude commented on Oct 23, 2026

jayzhan211 commented on Oct 23, 2026

mkleen commented on Oct 28, 2026

jayzhan211 commented on Oct 29, 2026

mkleen commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[EPIC] Handle `-0.0` (and NaN) float keys in one place instead of per call site #26138