[lake/iceberg] Support composite bucket keys for Iceberg-tiered tables #4936

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Fluss supports composite bucket keys ('bucket.key' = 'a,b'), and lookups depend on them: a prefix lookup must use exactly the bucket key, and delta joins look up the other side by it. Tables tiered to Iceberg are limited to a single bucket-key column, so users have to choose between composite bucket keys and Iceberg tiering.

In practice:

• Multi-column join keys can only use one column for the lookup. For example, a delta join child ⋈ parent ON child.parent_id = parent.id AND child.tenant_id = parent.tenant_id can only bucket child by parent_id. The lookup returns every row for that one column, and the rest of the condition is applied afterwards.
• Tenant-scoped keys don't fit. Where ids are only unique per tenant, the natural key is (tenant_id, id), and a single-column bucket key can't express it.
• Cascades of joins have to key every level by one root column, instead of keeping each level's natural composite key.

The single-key assumption lives in several places:

• the IcebergLakeCatalog table-creation check;
• IcebergPartitionSpecUtils, which builds a single bucket(key, N) transform;
• IcebergKeyEncoder / IcebergKeyDecoder, which require exactly one key field.

As explained in #4794, the restriction keeps each Fluss bucket aligned with exactly one Iceberg bucket. That alignment lets tiering be enabled later without redistributing data, and it makes bucket-level union reads and historical subscriptions efficient. Iceberg's bucket transform takes a single column, and several transforms (bucket(N, a), bucket(N, b)) produce a cross product rather than one of the N Fluss buckets.

Related work

• Iceberg tables with multiple bucket keys can be created but cannot be written #4794 / [Server] Prevent creating iceberg format tables with multiple bucket keys #4795: reject multi-key Iceberg tables at creation time. The right short-term fix, but it keeps the restriction.
• [lake/iceberg] Support multi-bucket-key tables #4069 / [lake/iceberg] Support multi-bucket-key and non-string partition keys #4070: remove the creation checks and use bucket(firstKey, N). This leaves IcebergKeyEncoder single-key, so writes still fail as in Iceberg tables with multiple bucket keys can be created but cannot be written #4794. Its partition spec also wouldn't match where rows actually land (the Fluss composite bucket), so pruning on a first-key filter could skip matching files. This issue proposes a different approach and could supersede [lake/iceberg] Support multi-bucket-key tables #4069.

Solution

Following the direction suggested in #4794: materialize the Fluss-computed composite bucket id in Iceberg and use it for bucket alignment and pruning, keeping today's layout for single-key tables.

Points to design:

1. Bucketing for composite keys in the Iceberg format: encode multi-field keys so the client, the tablet servers and tiering all compute the same bucket.
2. Iceberg layout for multi-key tables: for example the composite bucket id as an identity partition value, so Fluss bucket N maps to exactly one Iceberg partition. External engines keep correct results but lose key-based bucket pruning on these tables.
3. Read path: the split planner must extract the bucket from that layout, so union reads and historical subscriptions keep bucket-level pruning.
4. Compatibility: single-key tables unchanged; consistent with the FIP-27 clean table layout; enabling tiering on an existing multi-key table without redistributing data.

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!