Repository navigation
A retried upsert inserts a key that a concurrent writer already inserted #4194
Description
Apache Iceberg version
0.12.0 (latest release)
Please describe the bug 🐞
Currently, a commit retry replays the upsert's append, and _FastAppendFiles._validate_concurrency checks nothing, so a key another writer inserted after the read is inserted a second time. 0.11.1 raises CommitFailedException here.
import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField, StringType
cat = SqlCatalog("c", uri="sqlite:///:memory:", warehouse=f"file://{tempfile.mkdtemp()}")
cat.create_namespace("ns")
schema = Schema(NestedField(1, "k", LongType(), required=True), NestedField(2, "v", StringType()), identifier_field_ids=[1])
t = cat.create_table("ns.t", schema=schema)
row = lambda k, v: pa.Table.from_pylist([{"k": k, "v": v}], schema=schema.as_arrow())
t.append(row(1, "a"))
a = cat.load_table("ns.t")
b = cat.load_table("ns.t")
b.upsert(row(2, "b"))
a.upsert(row(2, "a"))
# Commit failed due to a concurrent update, retrying (1/4) in 106 ms
print(sorted(cat.load_table("ns.t").scan().to_arrow().to_pylist(), key=str))
# [{'k': 1, 'v': 'a'}, {'k': 2, 'v': 'a'}, {'k': 2, 'v': 'b'}], should hold k 2 onceTwo writers calling add_files on the same file register it twice the same way, and every row in it reads back twice.
tested on pyiceberg 0.12.0 and main
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity
sameer-malve commentedon Nov 3, 2026
Hi, I'd be interested in working on this issue.
I reproduced the problem on current main (
e3564a5) using Python 3.13 and a SQLite-backedSqlCatalog.When two concurrent upserts insert the same new key, both succeed after the retry, leaving duplicate rows. I also verified that retries involving different new keys work correctly, while concurrent updates to an existing key already raise
ValidationException.From the code investigation, the insert half of
Transaction.upsertusesself.append(), which goes through_FastAppendFiles. Its concurrency validation currently doesn't check for conflicting inserts during a retry.My proposed approach is to pass a conflict-detection filter built from the inserted rows' join keys and reuse the existing added-data-file validation. Ordinary appends would remain unchanged.
With Claude Code's assistance, an in-memory prototype was developed and tested. It prevented the duplicate under serializable isolation, preserved safe retries for non-conflicting keys, and passed the existing 162 relevant tests. Claude Code ran these additional prototype tests; I personally verified the original bug reproduction. The fix has not yet been applied to the repository.
Before implementing the fix, I'd appreciate guidance on two points:
write.update.isolation-levelsetting so both halves of an upsert follow the same isolation level, or would a separate merge isolation setting be preferable?_conflict_detection_filterparameter onTransaction.appendbe an acceptable way to pass the filter?The approach may conservatively reject some non-conflicting writes when file-level statistics overlap or are missing.
I'd be happy to prepare a focused PR with regression tests once we agree on the design.