Store: overlapping write batches from two runners are accepted, leaving duplicate entity versions (head check only looks at the last block of a batch) #6745

Summary

When a deployment is reassigned while its first write batch is still pending, the old and the new
runner can both write the same block range. deployment::transact_block accepts the second batch
because it only compares the last block of the batch against the deployment head. The result
is two versions of the same entity covering the same blocks. Queries then return the entity twice
(e.g. a singleton Protocol(id: "1") comes back as [{id:"1"},{id:"1"}]), while the PoI still
matches other indexers, because the PoI is computed from the stream of entity changes, not from
the stored rows.

Schemas with specVersion >= 1.3.0 are protected by accident: their deterministic vid
(block << 32 | seq) makes the second batch collide on the vid primary key. Schemas with
specVersion < 1.3.0 (vid bigserial) are not protected, and in release builds there is no
exclusion constraint on (id, block_range) (relational/ddl.rs, debug builds only).

Reproduced on master at 190f0aa (2026-10-19), graph-node 0.45.0 release build.

Mechanism

1. Runner A (node-a) starts a new deployment and accumulates a write batch (batching is on while
   the deployment is behind the chain head; GRAPH_STORE_WRITE_BATCH_DURATION /
   GRAPH_STORE_WRITE_BATCH_SIZE).
2. The deployment is reassigned to node-b before A commits. Runner B reads the head (still the
   start pointer) and starts processing from the start block.
3. A is unassigned; stop() pushes Request::Stop, and the queue commits A's pending batch
   [start..X_A].
4. B commits [start..X_B] with X_B > X_A. transact_block
   (store/postgres/src/deployment.rs, the block_ptr_from.number >= ptr.number check) compares
   X_B to the head X_A and accepts it.
5. Every entity written in [start..X_A] now has two versions per block. Entities that B never
   writes again after its first batch keep two live versions at the head.

The in-memory check added in #5419 lives in each process's WritableStore, so it cannot see a
second process. The writables cache in SubgraphStore::writable makes the race impossible
inside one process, but two graph-node processes sharing a database (the normal setup with
several index nodes) are not covered.

Reproduction

1. Store-level test (deterministic)

Branch with the test and the two-node recipe:
https://github.com/ellipfra/graph-node/tree/repro/overlapping-batches (two commits on top of
master 190f0aa, no production code changed).

The test store/test-store/tests/postgres/overlapping_batches.rs opens two
independent Store instances via StoreBuilder on the same database, simulating two
processes. For a specVersion 1.2.0 schema with one mutable entity Protocol updated at every
block:

• writer A and writer B are both obtained before any commit;
• A submits blocks 1..3, B submits blocks 1..5;
• A flushes, then B flushes.

Result (exclusion constraint replaced by the release-build gist index):

A flush: OK                     head = 3
B flush: OK                     head = 5
rows for id '1':
  vid=1 [1,2)  vid=2 [2,3)  vid=3 [3,)      <- A
  vid=4 [1,2)  vid=5 [2,3)  vid=6 [3,4)  vid=7 [4,5)  vid=8 [5,)   <- B
live versions of id '1': 2
list query at head: "1" returned twice

Controls: one writer writing 1..5 leaves one live version; with specVersion 1.3.0, B's flush
fails with duplicate key value violates unique constraint "protocol_pkey"; with the debug
exclusion constraint, it fails on protocol_id_block_range_excl.

2. Two real graph-node processes

Scripts, subgraph and step-by-step instructions: docs/repro/overlapping-batches/ on the same
branch. Setup: anvil with 21,000 pre-mined blocks; one subgraph (specVersion 0.0.4) with a single
unfiltered block handler that writes Protocol{id:"1", n: block}; two graph-node processes
(--node-id node-a, --node-id node-b --disable-block-ingestor) on one Postgres and one IPFS;
GRAPH_STORE_WRITE_BATCH_DURATION=60.

Natural race, no signals: subgraph_create + subgraph_deploy to node-a, then
subgraph_reassign to node-b 1 second later. Duplicated history in 3 out of 3 runs, no
error logged, deployment healthy. A committed its batch while stopping (~5,200 blocks), then B
committed [1..12121], which was accepted. Reassigning after 5, 15 or 30 s produced no duplicates
(A had already committed). In one 5 s run, B failed instead with a non-deterministic error:

writing Protocol entities at block 12121 failed: range lower bound must be less than or equal
to range upper bound Query: clamp ids [1]

That is the same race with the commit order reversed (B read the head before A's later commits).

Forced ordering (to make it deterministic): deploy to node-a, kill -STOP node-a while its
first batch is pending (head still at the start pointer, nothing committed), subgraph_reassign
to node-b, then kill -CONT node-a. node-a logs Terminating the subgraph runner because subgraph was unassigned and commits block_count: 2919. node-b then commits block_count: 12179, which
is accepted:

{"data":{"protocols":[{"id":"1","n":"2919"},{"id":"1","n":"12179"}]}}

Here the live duplicate disappears as soon as B overwrites the entity again, but the history stays
doubled: protocols(block: {number: 2000}) returns "1" twice.

Seen in production

On one operator's fleet, 12 deployments out of ~6,000 had this exact pattern: two full passes over
the first block range, starting at the manifest start block, all with specVersion < 1.3.0, all
created between 2023-09 and 2024-08. During that period, indexer-agent issued
subgraph_reassign right after subgraph_deploy (graphprotocol/indexer#920, #867/#868, #497).
None were created after the agent stopped doing that, although thousands of later deployments
still use specVersion < 1.3.0. A graphman copy of an affected deployment reproduces the
duplicates faithfully, because the copy carries the rows over. #5282 ("duplicate entity … in result
set") may be the same defect seen from the indexing side.

Related

• [Bug] Subgraph fails with non-deterministic error duplicate entity in result set #5282 (open): non-deterministic duplicate entity … in result set while indexing. This is
  what find_many reports when the store holds two live versions of an id, i.e. a likely
  consequence of this bug.
• Do not allow multiple active runners for a subgraph #5715 / Ensure that non-deterministic failures are cleared when a subgraph restarts after shutting down the writer too early #5452 (merged 2024-11): prevents two active runners for a subgraph within one
  process (restart during a retry delay). The reproduction above runs on a master that
  includes it: the two runners here live in different processes.
• [Bug] Reassigning subgraph to non-existent indexer doesn't stop indexing #5011 (closed): reassigning or pausing did not stop the old runner on another node (0.33).
  That is another way to end up with two writers on one deployment.
• [Bug] Subgraph hit error: internal constraint violated: Batches must go forward. Can't append a batch with block pointer... #4770 (open): Batches must go forward, raised by the in-memory check of a single process.

The visible effect is wrong query responses with a matching PoI. This has already led to a
conflicting-attestation report against an affected indexer.

Suggested fix

• In deployment::transact_block, reject a batch whose first block is not above the current
  head (Batch::first_block is already available), instead of checking only its last block. Returning
  DuplicateBlockProcessing there would turn both symptoms above (silent duplicates, and the
  clamp failure) into a clean, retryable refusal for the stale runner.
• Optionally, when a runner is stopped because it was unassigned, drop its pending batch instead
  of committing it. The batch belongs to a runner that no longer owns the deployment.
• Possibly a graphman check that detects overlapping live ranges for vid bigserial tables, so
  operators can find already-affected deployments.

Acknowledgements

Many thanks to @cargopete, who found this by comparing query attestations across indexers, wrote
it up in reproducible detail (requests, attestations, matching PoIs), and chose to report it as a
software bug rather than file a dispute. That report is what led us to the store-side
investigation and the reproduction above.

Environment

• graph-node master 190f0aa (0.45.0), release build; Postgres 17.10; anvil 1.8.5;
  graph-cli 0.97.1.