Skip to content

Bolt5002RoutingTableIT: DatabaseAreNotIdentical recurs 9 days after #5728's fix, with a different-looking root cause (bootstrap snapshot race, not replication skip) #6063

Description

@lvca

Follow-up observation from PR #6056's CI (unrelated engine change, integration-tests lane): Bolt5002RoutingTableIT.neo4jSchemeRoutesReadsAndWrites failed with the same symptom #5728 was filed and closed for on 2026-08-02 (fixed by #5743, "a schema change no longer skips replication when another thread holds the recording session").

Evidence

CI run 31462501377, integration-tests, 2026-08-11:

[ERROR] com.arcadedb.bolt.Bolt5002RoutingTableIT.neo4jSchemeRoutesReadsAndWrites -- Time elapsed: 11.49 s <<< FAILURE!
org.opentest4j.AssertionFailedError:

expected: 1L
 but was: 0L
	at com.arcadedb.bolt.Bolt5002RoutingTableIT.neo4jSchemeRoutesReadsAndWrites(Bolt5002RoutingTableIT.java:140)
	Suppressed: com.arcadedb.database.DatabaseComparator$DatabaseAreNotIdentical: Types: DB1 6 <> DB2 5
		at com.arcadedb.database.DatabaseComparator.compareTypes(DatabaseComparator.java:129)
		at com.arcadedb.database.DatabaseComparator.compare(DatabaseComparator.java:49)
		at com.arcadedb.server.BaseGraphServerTest.checkDatabasesAreIdentical(BaseGraphServerTest.java:560)
		at com.arcadedb.server.ha.raft.BaseRaftHATest.checkDatabasesAreIdentical(BaseRaftHATest.java:253)
		at com.arcadedb.server.BaseGraphServerTest.endTest(BaseGraphServerTest.java:236)

That Types: DB1 6 <> DB2 5 is the exact divergence #5728 documented. But the surrounding log context looks like a different mechanism than what #5743 fixed:

06:11:37.905 INFO  [Bolt5002RoutingTableIT] <ArcadeDB_2> Raft leader elected on server 2
06:11:37.927 INFO  [DriverFactory] Driver instance ... created for server uri 'neo4j://localhost:57699'
06:11:38.102 INFO  [BootstrapElection] Bootstrap source for 'graph': peer=localhost_2436, lastTxId=7, fingerprint=c9eca4af...b1f97471
06:11:38.128 SEVER [ArcadeStateMachine] Database 'graph': local lastTxId=8 is GREATER than cluster bootstrap lastTxId=7. This peer's data is fresher than the cluster's chosen baseline (committed BOOTSTRAP_FINGERPRINT_ENTRY). Refusing to overwrite local data. To preserve it, stop the cluster, copy this peer's database directory to every other peer, then restart all peers.
06:11:38.134 INFO  [ArcadeStateMachine] Database 'graph' bootstrap mismatch (local lastTxId=7 / fp=0d532652..., baseline lastTxId=7 / fp=c9eca4af...); reinstalling from leader-shipped full snapshot
06:11:38.217 INFO  [ArcadeStateMachine] Database 'graph' reinstalled after bootstrap mismatch
06:11:40.919 INFO  [Bolt5002RoutingTableIT] <ArcadeDB_2> END OF THE TEST: Check DBS are identical...

This is the third Raft leader election within the same test method (the two earlier elections in the class's other test methods completed and tore down cleanly - END OF THE TEST logged with no failure both times, at 06:11:16 and 06:11:29). This third election triggers a peer to detect a bootstrap/lastTxId mismatch and reinstall from a full leader-shipped snapshot - completing at 06:11:38.217, only ~2.7 s before the test's teardown comparator runs at 06:11:40.919. That is a much tighter window than "a schema change skips replication because another thread holds the recording session" (#5743's fix) - it looks like a post-snapshot-reinstall catch-up race: the routed write and/or the teardown comparison can run before a freshly-reinstalled follower has replayed everything the snapshot didn't cover.

Why this might not be a simple recurrence

Where to look first

Hypothesis, not a diagnosis (matching #5728's own framing): after ArcadeStateMachine finishes reinstalling a database from a snapshot (06:11:38.217), is there a synchronization point that guarantees the reinstalled peer has caught up on log entries after the snapshot baseline before it's considered readable/comparable? If the reinstall marks the peer ready as soon as the snapshot applies, but entries committed between the snapshot's lastTxId and the leader's current position are still in flight, a read or a checkDatabasesAreIdentical() landing in that window would see exactly this: fewer types on one peer, a 0L read that a moment later would resolve to 1L.

Also worth checking whether this is the same root cause behind the third election happening at all mid-test - two clean elections already occurred in the same suite before this one, so whatever forces ArcadeDB_2 to re-bootstrap on the third round is itself worth understanding (a restart? a leadership flap? something the test itself triggers as part of exercising routing-table refresh?).

Reproduction

Not reliably reproducible in isolation, same as #5728 - this is load/timing-sensitive on a shared CI runner. ha-integration-tests (a related but separate CI lane) is independently known to be chronically flaky on Raft/HA timing issues, so this may be one more symptom of that same family of races rather than something specific to the Bolt module.

Not something I investigated further - flagging with the evidence rather than assuming it's confirmed either as a regression of #5728 or as a wholly new bug.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions