[Bug] array_distinct / array_union / array_except do not canonicalize NaN like Spark

Describe the bug

Spark's array_distinct, array_union, and array_except use SQLOpenHashSet.withNaNCheckFunc to canonicalize NaN equality: every NaN bit pattern collapses to a single bucket, and +0.0 / -0.0 are treated as distinct. DataFusion's array_distinct, array_union, and array_except use hash-based set logic where NaN is not equal to NaN (IEEE semantics) and +0.0 == -0.0.

So for Float/Double element arrays containing NaN or signed zero, the results diverge silently:

array_distinct(array(double('NaN'), double('NaN'))) returns one element in Spark, two in Comet.
array_union(array(0.0), array(-0.0)) returns one element in Spark, two in Comet.

Surfaced by the array-expressions audit (collection PR queue).

Steps to reproduce

SELECT array_distinct(array(CAST('NaN' AS DOUBLE), CAST('NaN' AS DOUBLE)));
-- Spark: [NaN]      Comet: [NaN, NaN]

SELECT array_union(array(0.0), array(-0.0));
-- Spark: [0.0]      Comet: [0.0, -0.0]

Expected behavior

Either canonicalize NaN/signed-zero on the Comet side before invoking the DataFusion set operations, or downgrade array_distinct / array_union / array_except to Incompatible(Some(...)) whenever the element type is FloatType or DoubleType. (array_intersect already has a related Incompatible flag for ordering; the NaN gap is separate.)

Additional context

Comet serdes: CometScalarFunction("array_distinct") (registered in QueryPlanSerde.scala), CometArrayUnion and CometArrayExcept in spark/src/main/scala/org/apache/comet/serde/arrays.scala.
Spark reference: SQLOpenHashSet.withNaNCheckFunc and ArraySetLike in collectionOperations.scala.
Related: array_intersect carries an ordering Incompatible flag but does not document the NaN gap either.

DF follows DuckDb concepts which returns NaN, NaN, however I would suppose this is a DuckDB bug

D select array_distinct(array['nan'::float, 'nan'::float]);
┌─────────────────────────────────────────────────────────────────────┐
│ array_distinct((ARRAY[CAST('nan' AS FLOAT), CAST('nan' AS FLOAT)])) │
│                               float[]                               │
├─────────────────────────────────────────────────────────────────────┤
│ [nan, nan]                                                          │
└─────────────────────────────────────────────────────────────────────┘

However checking distincts without arrays give correct result, and arrays are not consistent, which IMO is a bug in duckDB and DF should follow more Spark semantics

D select 'nan'::float union select 'nan'::float;
┌──────────────────────┐
│ CAST('nan' AS FLOAT) │
│        float         │
├──────────────────────┤
│                  nan │
└──────────────────────┘

D select distinct a from (select 'nan'::float a union all select 'nan'::float);
┌───────┐
│   a   │
│ float │
├───────┤
│   nan │
└───────┘
D

DF resolves NaN correctly so its more Comet trouble

> SELECT array_distinct(make_array(CAST('NaN' AS DOUBLE), CAST('NaN' AS DOUBLE)));
+-----------------------------------------------------+
| array_distinct(make_array(Utf8("NaN"),Utf8("NaN"))) |
+-----------------------------------------------------+
| [NaN]                                               |
+-----------------------------------------------------+
1 row(s) fetched. 
Elapsed 0.013 seconds.

-0, +0 comparison is confirmed to be DF problem

Navigation Menu

Repository navigation

[Bug] array_distinct / array_union / array_except do not canonicalize NaN like Spark #4481

Describe the bug

Steps to reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug] array_distinct / array_union / array_except do not canonicalize NaN like Spark #4481

Description

Describe the bug

Steps to reproduce

Expected behavior

Additional context

Activity

comphead commented on Jun 3, 2026

comphead commented on Jun 5, 2026

comphead commented on Jun 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions