Skip to content

Native Iceberg writes accept write.parquet.compression-codec=none, which iceberg-java rejects #7008

Open

Description

@andygrove

Describe the bug

The native Iceberg write gate admits write.parquet.compression-codec=none, and the native writer writes the table uncompressed. iceberg-java rejects that value. Parquet.WriteBuilder resolves the codec with CompressionCodecName.valueOf(codec.toUpperCase(Locale.ENGLISH)), and parquet-mr's CompressionCodecName has UNCOMPRESSED but no NONE, so the write fails with Unsupported compression codec: none. With native Iceberg writes enabled, a write that fails without Comet succeeds with it.

The mapping is deliberate. IcebergWriteProtoTranslation.resolveCompression maps none to uncompressed, its scaladoc says Iceberg accepts none, and IcebergWriteProtoTranslationSuite pins the mapping. none is ORC's spelling. Iceberg documents uncompressed for Parquet.

Three more settings pass the gate because it trims or lowercases them, while iceberg-java reads them as written:

  • The codec itself. resolveCompression trims and lowercases it, so ' gzip' writes natively.
  • write.parquet.row-group-check-min-record-count and write.parquet.row-group-check-max-record-count. requireIntPropertyAtDefault trims them, so ' 100' passes as the default. iceberg-java reads them with Integer.parseInt.
  • write.parquet.page-version on Iceberg 1.11. requireParquetPageVersionDefault trims and lowercases it, so V1 passes. parquet-mr's WriterVersion.fromString accepts only v1, v2 or the enum names. Earlier Iceberg versions have no such property.

The native files are ordinary Parquet, so readers are not affected. The cost is that the same table accepts some writes and fails others: a write that falls back to iceberg-java, for example one whose input does not run in Comet, fails on the setting a native write accepted.

Steps to reproduce

iceberg-java's side was run with a single-file Java program that calls Parquet.write(Files.localOutput(f)).set(key, value).createWriterFunc(...).build() and writes one row, against iceberg-spark-runtime-4.1_2.13:1.11.0 and iceberg-spark-runtime-3.4_2.12:1.5.2:

write.parquet.compression-codec='none'                 IllegalArgumentException: Unsupported compression codec: none
write.parquet.compression-codec=' gzip'                IllegalArgumentException: Unsupported compression codec:  gzip
write.parquet.row-group-check-min-record-count=' 100'  NumberFormatException: For input string: " 100"
write.parquet.page-version='V1' (1.11.0 only)          IllegalArgumentException: Unsupported Parquet page version: V1 (must be v1 or v2)

uncompressed, GZIP and v1 write successfully in the same program.

Comet's side is read from the code and was not run end to end. No rule in CometIcebergNativeWrite looks at the codec name, so with spark.comet.write.iceberg.enabled=true and spark.comet.exec.localTableScan.enabled=true:

CREATE TABLE local.db.t (id bigint) USING iceberg
  TBLPROPERTIES ('write.parquet.compression-codec'='none');
INSERT INTO local.db.t VALUES (1), (2);  -- expected to plan CometIcebergWrite and write uncompressed files

Without Comet, the INSERT fails with Unsupported compression codec: none.

Expected behavior

The gate should decline a write whose codec or other setting iceberg-java would reject, so that it falls back and fails as it does without Comet. For the codec, that means admitting only the names iceberg-java accepts, compared after upper-casing and without trimming. For the record counts and the page version, it means parsing them the way iceberg-java does. The none case in IcebergWriteProtoTranslationSuite would change with it.

Additional context

Found while auditing the native Iceberg write docs against the code. #6960 is a related case: there, a numeric gzip level that iceberg-java fails on was listed as an accepted divergence rather than reproduced, because the level is a documented Iceberg setting that Hadoop parses as an enum name. None of the values here is a valid iceberg-java setting, so falling back keeps iceberg-java's behavior and its error.

Activity

  1. added 2 commits that reference this issue on Oct 15, 2026
    79fe8e5
    eefe173
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:Icebergarea:writerNative Parquet writerbugSomething isn't workingpriority:lowMinor issues, test failures, tooling, cosmetic

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions