Repository navigation
add_files puts a file holding a value and NULLs in that value's partition #4191
Description
Apache Iceberg version
0.12.0 (latest release)
Please describe the bug 🐞
Currently, add_files takes an identity partition from the column's min and max and ignores its null count, so a file holding 'a' and NULL is registered as partition 'a' and IS NULL filters prune it. A file holding 'a' and 'b' is refused.
import tempfile
import pyarrow as pa, pyarrow.parquet as pq
from pyiceberg.catalog.sql import SqlCatalog
d = tempfile.mkdtemp()
cat = SqlCatalog("c", uri="sqlite:///:memory:", warehouse=f"file://{d}")
cat.create_namespace("ns")
df = pa.table({"x": [1, 2], "s": ["a", None]})
t = cat.create_table("ns.t", schema=df.schema)
with t.update_spec() as u:
u.add_identity("s")
pq.write_table(df, f"{d}/a.parquet")
t.add_files([f"{d}/a.parquet"])
print(t.scan(row_filter="s IS NULL").to_arrow()["x"].to_pylist())
# [], should be [2]
pq.write_table(pa.table({"x": [3, 4], "s": ["a", "b"]}), f"{d}/b.parquet")
t.add_files([f"{d}/b.parquet"])
# ValueError: Cannot infer partition value from parquet metadata as there are more than one partition values ...day, truncate and a NaN beside 1.0 do the same, and with write.metadata.metrics.default set to none every added file gets a NULL partition.
tested on pyiceberg 0.12.0 and main
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity