Iceberg truncate on integers with widths above 2^30: native kernel and native writer compute different partition values #7442

Describe the bug

For an integer truncate[W] with W > 2^30, Comet computes two different values for the same row.

• The system.truncate kernel reproduces Iceberg Java's TruncateUtil (truncate.rs). In Java, (v % W) + W overflows int for these widths. tinyint and smallint inputs go through the same function.
• The native writer gets partition values from iceberg-rust's PartitionSplitter (iceberg_write.rs). At the pinned revision, iceberg-rust computes v - v.rem_euclid(width), which never overflows.

For W = 2_000_000_000 and v = 1_500_000_000, the kernel returns -2_000_000_000 and iceberg-rust returns 0. apache/iceberg-rust#3141 reports this mismatch as the clustered writer's "The input is not sorted!" error.

The iceberg-rust value is the one the spec requires. The truncate transform details define integer truncation as v - (v % W) with a positive remainder. Java's overflow makes the remainder negative. Iceberg Java treats this as a bug in apache/iceberg#18111, with a fix in apache/iceberg#18112 that isn't merged yet. The iceberg-rust discussion is in apache/iceberg-rust#3141 and apache/iceberg-rust#3573.

The comment on truncate_agrees_with_iceberg_rust calls the large-width case an iceberg-rust bug. Per the spec it is a Java bug. Only the i32::MIN and i64::MIN cases are iceberg-rust bugs: there the result doesn't fit in the type, the spec is silent, Java wraps, and iceberg-rust panics in debug builds. The user guide describes the kernels as reproducing Java's semantics and doesn't mention the spec.

Steps to reproduce

Add a large-width case to truncate_agrees_with_iceberg_rust in native/core/src/execution/operators/iceberg_write.rs, for example Int32Array::from(vec![Some(1_500_000_000)]) with width 2_000_000_000. The kernel and iceberg-rust's transform disagree (-2000000000 against 0).

Expected behavior

Comet should match Spark, and Spark's result here depends on the Iceberg jar the user runs, not on the Spark version. Every Iceberg release through 1.12.0 overflows. A release with apache/iceberg#18112 will return the spec value. A Spark-version shim can't pick the right behavior, because users choose their Iceberg runtime independently of the Spark profile.

Comet should fall back to Spark in two cases:

• An integer truncate call with W > 2^30, so the result is whatever the user's Iceberg jar computes.
• A native Iceberg write whose partition spec has an integer truncate[W] with W > 2^30. Native writes produce the spec value, while Spark on Iceberg 1.12.0 and earlier writes the overflowed value into the same table.

Comet doesn't control which Iceberg jar users bring, so this fallback stays after Iceberg ships the fix. If these widths ever need to run natively, Comet could probe the jar at plan time instead: TruncateUtil.truncateInt(Integer.MAX_VALUE, Integer.MAX_VALUE - 1) returns 0 when the jar follows the spec. Probing the behavior catches backports and vendor builds that parsing IcebergBuild.version() would miss. The kernel would then take a flag for spec or overflow semantics, and native writes would run only when the jar follows the spec.

The test comment and the user guide should also cite the spec. The large-width overflow is a Java bug. The wrap at the integer minimum is Java's choice where the spec is silent.

Additional context

Widths above 2^30 are rare in practice. The fallback doesn't affect other widths, because every width up to 2^30 gives the same value under both formulas.