[HA] Bolt5002RoutingTableIT: type created via Cypher-over-Bolt never replicates to all followers

[robfrank]

Split out of #5630, which classified Bolt5002RoutingTableIT.neo4jSchemeRoutesReadsAndWrites as an "unsettled replication" flake wanting a bounded poll. That classification is wrong - the state never settles, so this is a real defect and not a test-hardening problem.

Evidence

PR #5704 added a poll over every started node, waiting up to 60 s for the Bolt5002Route type and its single record after the routed write. CI (run 30706875826, integration-tests):

org.awaitility.core.ConditionTimeoutException:
  Assertion condition defined as a Lambda expression in com.arcadedb.bolt.Bolt5002RoutingTableIT
  [server 1 must have replicated type Bolt5002Route]
  Expecting value to be true but was false within 1 minutes
    at Bolt5002RoutingTableIT.awaitTypeReplicatedOnEveryNode(Bolt5002RoutingTableIT.java:187)
    at Bolt5002RoutingTableIT.neo4jSchemeRoutesReadsAndWrites(Bolt5002RoutingTableIT.java:140)

  Suppressed: com.arcadedb.database.DatabaseComparator$DatabaseAreNotIdentical:
    Types: DB1 6 <> DB2 5
      at DatabaseComparator.compareTypes(DatabaseComparator.java:129)
      at BaseGraphServerTest.checkDatabasesAreIdentical(BaseGraphServerTest.java:560)

Two independent signals for the same divergence:

1. Server 1 does not have the type after 60 seconds of polling. Not lag.
2. The teardown byte-for-byte comparison reports the follower is one type behind - the same Types: DB1 6 <> DB2 5 that [Misc] Flaky CI tests: wall-clock budgets, short read timeouts, and unsettled counters #5630 recorded as a suppressed error.

The poll was reverted in #5704 (the file there is byte-identical to main); polling would only have turned a fast failure into a slow one.

Reproduction

Bolt5002RoutingTableIT.neo4jSchemeRoutesReadsAndWrites. The write is a Cypher CREATE (n:Bolt5002Route {name: 'routed'}) issued through a neo4j:// driver session against the leader, which implicitly creates the vertex type. It passes when run in isolation (#5630 reports 3/3) and fails under a loaded CI runner, so it is load- or timing-sensitive rather than deterministic.

Where to look first

Hypothesis, not a diagnosis. This has the shape of the #5492 / #5655 family: code that commits while holding the inner LocalDatabase instead of the wrapped replicated instance applies its pages locally and replicates nothing, and nothing throws at the point of the mistake.

ha-raft/CLAUDE.md documents the rule and getWrappedDatabaseInstance() as the always-correct resolution. #5655 fixed exactly this for Cypher statements (commit 7469a24) and #5492 for SQL (e64160a) - both are already in main, so if this is the same family it is a path those did not cover. The implicit type creation on the Bolt write path is worth checking specifically, since the record and the schema entry may not travel the same route.

Worth confirming before assuming: whether the record replicates while only the type does not.

[robfrank]

Important qualifier for whoever picks this up: the failure is intermittent, not deterministic.

On the very next CI run (30718297804, commit b126b7d6a, the revert), the unmodified test passed 3/3 in 50.45 s. The failing run was 30706875826.

This does not weaken the finding - when the divergence occurs it is unambiguous, since a 60 s poll never sees the type and the teardown comparator independently reports Types: DB1 6 <> DB2 5. But it does mean:

• Do not expect a clean reproduction. Like the original [Misc] Flaky CI tests: wall-clock budgets, short read timeouts, and unsettled counters #5630 symptoms, it appears under a loaded shared runner and not in isolation.
• A green run is not evidence the defect is absent, so "I ran it and it passed" should not close this.
• The most reliable signal is the teardown DatabaseAreNotIdentical rather than the routed read, since the read can be answered by whichever follower happens to have caught up.

The load-sensitivity is itself a clue: a schema entry that replicates under light load and not under heavy load points at a race or an ordering dependency in how the type creation reaches followers, rather than a path that never replicates at all.