Follow-up observation from PR #6056's CI (unrelated engine change, integration-tests lane): Bolt5002RoutingTableIT.neo4jSchemeRoutesReadsAndWrites failed with the same symptom #5728 was filed and closed for on 2026-08-02 (fixed by #5743, "a schema change no longer skips replication when another thread holds the recording session").
Evidence
CI run 31462501377, integration-tests, 2026-08-11:
[ERROR] com.arcadedb.bolt.Bolt5002RoutingTableIT.neo4jSchemeRoutesReadsAndWrites -- Time elapsed: 11.49 s <<< FAILURE!
org.opentest4j.AssertionFailedError:
expected: 1L
but was: 0L
at com.arcadedb.bolt.Bolt5002RoutingTableIT.neo4jSchemeRoutesReadsAndWrites(Bolt5002RoutingTableIT.java:140)
Suppressed: com.arcadedb.database.DatabaseComparator$DatabaseAreNotIdentical: Types: DB1 6 <> DB2 5
at com.arcadedb.database.DatabaseComparator.compareTypes(DatabaseComparator.java:129)
at com.arcadedb.database.DatabaseComparator.compare(DatabaseComparator.java:49)
at com.arcadedb.server.BaseGraphServerTest.checkDatabasesAreIdentical(BaseGraphServerTest.java:560)
at com.arcadedb.server.ha.raft.BaseRaftHATest.checkDatabasesAreIdentical(BaseRaftHATest.java:253)
at com.arcadedb.server.BaseGraphServerTest.endTest(BaseGraphServerTest.java:236)
That Types: DB1 6 <> DB2 5 is the exact divergence #5728 documented. But the surrounding log context looks like a different mechanism than what #5743 fixed:
06:11:37.905 INFO [Bolt5002RoutingTableIT] <ArcadeDB_2> Raft leader elected on server 2
06:11:37.927 INFO [DriverFactory] Driver instance ... created for server uri 'neo4j://localhost:57699'
06:11:38.102 INFO [BootstrapElection] Bootstrap source for 'graph': peer=localhost_2436, lastTxId=7, fingerprint=c9eca4af...b1f97471
06:11:38.128 SEVER [ArcadeStateMachine] Database 'graph': local lastTxId=8 is GREATER than cluster bootstrap lastTxId=7. This peer's data is fresher than the cluster's chosen baseline (committed BOOTSTRAP_FINGERPRINT_ENTRY). Refusing to overwrite local data. To preserve it, stop the cluster, copy this peer's database directory to every other peer, then restart all peers.
06:11:38.134 INFO [ArcadeStateMachine] Database 'graph' bootstrap mismatch (local lastTxId=7 / fp=0d532652..., baseline lastTxId=7 / fp=c9eca4af...); reinstalling from leader-shipped full snapshot
06:11:38.217 INFO [ArcadeStateMachine] Database 'graph' reinstalled after bootstrap mismatch
06:11:40.919 INFO [Bolt5002RoutingTableIT] <ArcadeDB_2> END OF THE TEST: Check DBS are identical...
This is the third Raft leader election within the same test method (the two earlier elections in the class's other test methods completed and tore down cleanly - END OF THE TEST logged with no failure both times, at 06:11:16 and 06:11:29). This third election triggers a peer to detect a bootstrap/lastTxId mismatch and reinstall from a full leader-shipped snapshot - completing at 06:11:38.217, only ~2.7 s before the test's teardown comparator runs at 06:11:40.919. That is a much tighter window than "a schema change skips replication because another thread holds the recording session" (#5743's fix) - it looks like a post-snapshot-reinstall catch-up race: the routed write and/or the teardown comparison can run before a freshly-reinstalled follower has replayed everything the snapshot didn't cover.
Why this might not be a simple recurrence
Where to look first
Hypothesis, not a diagnosis (matching #5728's own framing): after ArcadeStateMachine finishes reinstalling a database from a snapshot (06:11:38.217), is there a synchronization point that guarantees the reinstalled peer has caught up on log entries after the snapshot baseline before it's considered readable/comparable? If the reinstall marks the peer ready as soon as the snapshot applies, but entries committed between the snapshot's lastTxId and the leader's current position are still in flight, a read or a checkDatabasesAreIdentical() landing in that window would see exactly this: fewer types on one peer, a 0L read that a moment later would resolve to 1L.
Also worth checking whether this is the same root cause behind the third election happening at all mid-test - two clean elections already occurred in the same suite before this one, so whatever forces ArcadeDB_2 to re-bootstrap on the third round is itself worth understanding (a restart? a leadership flap? something the test itself triggers as part of exercising routing-table refresh?).
Reproduction
Not reliably reproducible in isolation, same as #5728 - this is load/timing-sensitive on a shared CI runner. ha-integration-tests (a related but separate CI lane) is independently known to be chronically flaky on Raft/HA timing issues, so this may be one more symptom of that same family of races rather than something specific to the Bolt module.
Not something I investigated further - flagging with the evidence rather than assuming it's confirmed either as a regression of #5728 or as a wholly new bug.
Follow-up observation from PR #6056's CI (unrelated engine change,
integration-testslane):Bolt5002RoutingTableIT.neo4jSchemeRoutesReadsAndWritesfailed with the same symptom #5728 was filed and closed for on 2026-08-02 (fixed by #5743, "a schema change no longer skips replication when another thread holds the recording session").Evidence
CI run 31462501377,
integration-tests, 2026-08-11:That
Types: DB1 6 <> DB2 5is the exact divergence #5728 documented. But the surrounding log context looks like a different mechanism than what #5743 fixed:This is the third Raft leader election within the same test method (the two earlier elections in the class's other test methods completed and tore down cleanly -
END OF THE TESTlogged with no failure both times, at 06:11:16 and 06:11:29). This third election triggers a peer to detect a bootstrap/lastTxIdmismatch and reinstall from a full leader-shipped snapshot - completing at06:11:38.217, only ~2.7 s before the test's teardown comparator runs at06:11:40.919. That is a much tighter window than "a schema change skips replication because another thread holds the recording session" (#5743's fix) - it looks like a post-snapshot-reinstall catch-up race: the routed write and/or the teardown comparison can run before a freshly-reinstalled follower has replayed everything the snapshot didn't cover.Why this might not be a simple recurrence
LocalDatabaseinstance a commit lands on. Nothing in today's log suggests that: this run shows a peer that is replicating, but via a full-snapshot bootstrap reinstall triggered by an election, immediately followed by the identity check.Where to look first
Hypothesis, not a diagnosis (matching #5728's own framing): after
ArcadeStateMachinefinishes reinstalling a database from a snapshot (06:11:38.217), is there a synchronization point that guarantees the reinstalled peer has caught up on log entries after the snapshot baseline before it's considered readable/comparable? If the reinstall marks the peer ready as soon as the snapshot applies, but entries committed between the snapshot'slastTxIdand the leader's current position are still in flight, a read or acheckDatabasesAreIdentical()landing in that window would see exactly this: fewer types on one peer, a0Lread that a moment later would resolve to1L.Also worth checking whether this is the same root cause behind the third election happening at all mid-test - two clean elections already occurred in the same suite before this one, so whatever forces
ArcadeDB_2to re-bootstrap on the third round is itself worth understanding (a restart? a leadership flap? something the test itself triggers as part of exercising routing-table refresh?).Reproduction
Not reliably reproducible in isolation, same as #5728 - this is load/timing-sensitive on a shared CI runner.
ha-integration-tests(a related but separate CI lane) is independently known to be chronically flaky on Raft/HA timing issues, so this may be one more symptom of that same family of races rather than something specific to the Bolt module.Not something I investigated further - flagging with the evidence rather than assuming it's confirmed either as a regression of #5728 or as a wholly new bug.