Repository navigation
Description
Describe the bug
The native Iceberg write gate admits write.parquet.compression-codec=none, and the native writer writes the table uncompressed. iceberg-java rejects that value. Parquet.WriteBuilder resolves the codec with CompressionCodecName.valueOf(codec.toUpperCase(Locale.ENGLISH)), and parquet-mr's CompressionCodecName has UNCOMPRESSED but no NONE, so the write fails with Unsupported compression codec: none. With native Iceberg writes enabled, a write that fails without Comet succeeds with it.
The mapping is deliberate. IcebergWriteProtoTranslation.resolveCompression maps none to uncompressed, its scaladoc says Iceberg accepts none, and IcebergWriteProtoTranslationSuite pins the mapping. none is ORC's spelling. Iceberg documents uncompressed for Parquet.
Three more settings pass the gate because it trims or lowercases them, while iceberg-java reads them as written:
- The codec itself.
resolveCompressiontrims and lowercases it, so' gzip'writes natively. write.parquet.row-group-check-min-record-countandwrite.parquet.row-group-check-max-record-count.requireIntPropertyAtDefaulttrims them, so' 100'passes as the default. iceberg-java reads them withInteger.parseInt.write.parquet.page-versionon Iceberg 1.11.requireParquetPageVersionDefaulttrims and lowercases it, soV1passes. parquet-mr'sWriterVersion.fromStringaccepts onlyv1,v2or the enum names. Earlier Iceberg versions have no such property.
The native files are ordinary Parquet, so readers are not affected. The cost is that the same table accepts some writes and fails others: a write that falls back to iceberg-java, for example one whose input does not run in Comet, fails on the setting a native write accepted.
Steps to reproduce
iceberg-java's side was run with a single-file Java program that calls Parquet.write(Files.localOutput(f)).set(key, value).createWriterFunc(...).build() and writes one row, against iceberg-spark-runtime-4.1_2.13:1.11.0 and iceberg-spark-runtime-3.4_2.12:1.5.2:
write.parquet.compression-codec='none' IllegalArgumentException: Unsupported compression codec: none
write.parquet.compression-codec=' gzip' IllegalArgumentException: Unsupported compression codec: gzip
write.parquet.row-group-check-min-record-count=' 100' NumberFormatException: For input string: " 100"
write.parquet.page-version='V1' (1.11.0 only) IllegalArgumentException: Unsupported Parquet page version: V1 (must be v1 or v2)
uncompressed, GZIP and v1 write successfully in the same program.
Comet's side is read from the code and was not run end to end. No rule in CometIcebergNativeWrite looks at the codec name, so with spark.comet.write.iceberg.enabled=true and spark.comet.exec.localTableScan.enabled=true:
CREATE TABLE local.db.t (id bigint) USING iceberg
TBLPROPERTIES ('write.parquet.compression-codec'='none');
INSERT INTO local.db.t VALUES (1), (2); -- expected to plan CometIcebergWrite and write uncompressed filesWithout Comet, the INSERT fails with Unsupported compression codec: none.
Expected behavior
The gate should decline a write whose codec or other setting iceberg-java would reject, so that it falls back and fails as it does without Comet. For the codec, that means admitting only the names iceberg-java accepts, compared after upper-casing and without trimming. For the record counts and the page version, it means parsing them the way iceberg-java does. The none case in IcebergWriteProtoTranslationSuite would change with it.
Additional context
Found while auditing the native Iceberg write docs against the code. #6960 is a related case: there, a numeric gzip level that iceberg-java fails on was listed as an accepted divergence rather than reproduced, because the level is a documented Iceberg setting that Hadoop parses as an enum name. None of the values here is a valid iceberg-java setting, so falling back keeps iceberg-java's behavior and its error.
Activity
fix: fall back for Iceberg write settings that iceberg-java rejects
fix: fall back for Iceberg write settings that iceberg-java rejects