[lake/iceberg] Truncated position-delete bounds break compaction #4940

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

1.0.0 (latest release)

Please describe the bug 🐞

For primary-key tables tiered to Iceberg, Fluss writes position-delete files that keep only the first 16 characters of each data-file path (e.g. s3://data-lakeho). Iceberg can't tell which data file such a delete belongs to. So it treats every Fluss position delete as possibly applying to every data file in its partition, even though Fluss only deletes rows it wrote in the same commit.

This is visible in the delete_files metadata table: the lower and upper file_path bounds are different 16-character prefixes, and referenced_data_file is null.

Impact

• Compaction fails while tiering is running. Each tiering round adds position deletes to every partition. Any rewrite that commits after the next round fails with Cannot commit, found new position delete for replaced data file. With partial progress enabled, those groups are dropped silently and the job still reports success. In our setup (9 buckets, 5-minute tiering, hourly Spark rewrite_data_files), only 1–3 of 9 buckets were compacted per run, across 44 consecutive runs.
• Compaction rewrites whole partitions every run, because every data file appears to have deletes.
• Reads are slower, because every position-delete file is checked against every data file in its partition.

Root cause

The delta writer's GenericAppenderFactory is created without the table. Iceberg then uses its default metrics mode, which keeps only 16 characters, instead of the position-delete config that keeps full paths. Table properties can't work around it, because they're never passed to the factory.

Solution

Pass the table to GenericAppenderFactory in TaskWriterFactory, so position deletes keep full file_path bounds and Iceberg ties each one to a single data file.

Are you willing to submit a PR?

• I'm willing to submit a PR!