delete() and overwrite() raise Cannot create expression literal from NaN on a NaN identity partition #4277

Apache Iceberg version

main (development)

Please describe the bug 🐞

A NaN partition value prevents deleting or replacing the data file that contains it. Even deleting every row or overwriting the whole table fails. This is a regression in 0.12.0. The reproduction below needs main, because 0.12.0 cannot append a NaN partition.

I reproduced this on main at e3564a54c10960307147377c578680276865ed18, with Python 3.12 and PyArrow 25.0.1 on Linux/amd64. The table has an identity partition on a double column. Deletes also fail with a float column.

Install from main, for example pip install "pyiceberg[pyarrow,sql-sqlite] @ git+https://github.com/apache/iceberg-python@e3564a54c10960307147377c578680276865ed18".

The reproduction uses a SQLite catalog and a temporary warehouse. It appends two rows, one with a NaN partition value, then tries to delete that partition:

import tempfile

import pyarrow as pa

from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import DoubleType, IntegerType, NestedField

warehouse = tempfile.mkdtemp()
catalog = SqlCatalog("default", uri=f"sqlite:///{warehouse}/catalog.db", warehouse=f"file://{warehouse}")
catalog.create_namespace("default")

schema = Schema(
    NestedField(1, "id", IntegerType(), required=False),
    NestedField(2, "value", DoubleType(), required=False),
)
partition_spec = PartitionSpec(PartitionField(source_id=2, field_id=1000, transform=IdentityTransform(), name="value"))
tbl = catalog.create_table("default.nan_partition", schema=schema, partition_spec=partition_spec)

tbl.append(pa.Table.from_pydict({"id": [1, 2], "value": [1.0, float("nan")]}, schema=schema.as_arrow()))
print(tbl.scan().to_arrow().to_pylist())

tbl.delete("value is nan")

Output, with the traceback shortened:

[{'id': 1, 'value': 1.0}, {'id': 2, 'value': nan}]
Traceback (most recent call last):
  File "repro.py", line 25, in <module>
    tbl.delete("value is nan")
  ...
  File ".../pyiceberg/table/update/snapshot.py", line 541, in _build_delete_files_partition_predicate
    self._transaction._build_partition_predicate(
  File ".../pyiceberg/table/__init__.py", line 424, in _build_partition_predicate
    EqualTo(Reference(partition_field), partition_record[pos])
  ...
  File ".../pyiceberg/expressions/literals.py", line 98, in __init__
    raise ValueError("Cannot create expression literal from NaN.")
ValueError: Cannot create expression literal from NaN.

Expected: the delete succeeds and leaves only the row whose partition value is 1.0.

Actual: ValueError: Cannot create expression literal from NaN.

I also tested a four-row table with two rows in each partition:

• delete("value is nan"), which would drop the whole NaN file, and delete("id = 3"), which would rewrite it, both fail.
• delete(AlwaysTrue()) fails too.
• overwrite(df) fails even when the new data has no NaN, and so do overwrites filtered to the NaN file ("value is nan", "id = 3").
• upsert fails when updating a row in the NaN partition.
• dynamic_partition_overwrite fails when the incoming data contains a NaN partition value.

After each failure, reloading the table returns the same snapshot and rows, with no new metadata files. The failed rewrites and dynamic overwrite each leave one unreferenced data file. Operations that touch only non-NaN files succeed. Dynamic overwrite also succeeds when NaN is present only in an untouched partition.

_build_partition_predicate emits IsNull for a None value and EqualTo for any other value. Expression literals reject NaN, so that EqualTo raises. Two callers reach it. _build_delete_files_partition_predicate passes the partitions of the files being deleted. dynamic_partition_overwrite passes the partitions of the incoming data. A narrow fix would be to emit IsNaN for a NaN value, next to the existing IsNull branch.

#3011 added the snapshot path that builds partition predicates for manifest pruning. I also committed the NaN data file directly with fast_append instead of append. On that table, delete("value is nan"), delete(AlwaysTrue()) and overwrite(df) succeed on 0.11.1 but raise this error on 0.12.0. Tables whose NaN partitions were written by another engine should fail the same way, but I did not test another engine.

#4030 fixed writing NaN partitions and is not yet released. On 0.12.0, append raises ZeroDivisionError before reaching this bug. On 0.12.0 and 0.11.1, that error also hides the rewrite, upsert and dynamic-overwrite cases.

Java's ManifestFilterManager and BaseReplacePartitions collect deleted partitions in a set and match manifests against that set, so they never build an expression literal from a partition value. I read the Java source but did not run it.

#4130 is another NaN problem in delete(). There, missing NaN counts make the strict evaluator drop NaN rows. In the whole-file delete reproduced here, the partition predicate raises before anything is written.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time