[Feature] PIP-44: Data evolution for primary key tables #11064

Search before asking

• I searched in the issues and found nothing similar.

Motivation

A wide primary key table often needs to update a few columns, such as a historical backfill or a recomputed metric. Today that write stores another full primary-key record. partial-update can send only the changed fields, but the old values stay in older files until compaction and expiration.

PIP-44 proposes physical column-level updates by reusing Data Evolution alignment, with firstRowId as an internal alignment id rather than row tracking. The discussion covers this approach, and the proposer agreed that the work can proceed.

This issue tracks the implementation. The first pull request is the foundation only.

Solution

A primary key table can enable data-evolution.enabled without row-tracking.enabled.

• Newly written full rows receive an alignment range (firstRowId, rowCount) at commit, from Snapshot.nextRowId. No _ROW_ID column is added.
• A caller can commit a column file for one existing base file: same (partition, bucket, firstRowId, rowCount), non-key and non-partition columns only, level = 0, via compactIncrement (CommitKind.COMPACT).
• Reads stitch the column file onto the base file before the merge function runs. Sequence and value kind come from the base file, so a backfill does not become a newer record. A merge engine therefore sees an ordinary full row and needs no separate read path. For example, a null field in a newer partial-update record keeps the backfilled value and a non-null field replaces it; an aggregate such as max uses the backfilled value.
• Predicates are applied to the whole alignment group. A split that contains a column file is not raw-convertible. Limit pushdown, at both the manifest scan and the split plan, keeps that bucket.
• Compaction rewrites a base file together with its column files into full rows and assigns a new alignment range. Column files are not upgraded on their own.
• Commit keeps primary-key conflict detection and adds alignment checks: one base file per group, equal row counts, no overlapping ranges, and one file per column.
• An existing table can turn the option on (false to true only). Files written before that have no alignment range. They can be read until a column file appears in the same bucket; a full compaction of the bucket migrates them.
• Column-file rows are not added to totalRecordCount or partition recordCount.

Constraints for this version: fixed bucket (bucket > 0) or dynamic bucket (bucket = -1). Merge engines that do not require lookup are allowed; deduplicate, partial-update, and aggregation are covered by tests, and first-row is rejected because it requires lookup. A new merge engine is allowed unless it requires lookup. Changelog producer is none or input. No deletion vectors, no lookup, no postpone bucket (bucket = -2), no changelog-producer=full-compaction, no Iceberg metadata, and no row tracking. Dynamic bucket still uses the existing write API: the caller passes the bucket, and alignment stays per (partition, bucket). The read-optimized table is rejected, because column files stay at level 0 and reading only the top level would return the pre-backfill values.

There is no SQL entry point. The caller writes the column file through a Java API and must keep the base file's row order. The committer checks the row count only. A column file whose rows are in another order is committed and silently corrupts those columns.

PrimaryKeyDataEvolutionTableTest covers this version: validation, alignment assignment, backfill and replacement, compaction including a discarded compaction, both limit layers, whole-group predicate pruning, legacy-file migration, partial-update, aggregation, and dynamic bucket.

Anything else?

Follow-ups:

• MERGE INTO and locating rowIndex by primary key, so a backfill can be expressed without the caller managing row order.
• Column families (column-family.cols.*) and writing one logical row as multiple physical files.
• Compaction output that keeps column files instead of rewriting them into full rows.
• Deletion vectors, lookup, first-row, postpone bucket, changelog-producer=full-compaction, and Iceberg metadata.
• Python read, write, and validation.

The first pull request should use Part of #<this issue>. Fixes would close this issue before those follow-ups. Please assign this issue before that pull request is opened.

Are you willing to submit a PR?

• I'm willing to submit a PR!