ha-integration-tests fails on main ~75% of the time, with a different test each run

[lvca]

What happens

ha-integration-tests is failing on main most of the time, and it fails on a different test almost every run.
The last 8 runs of the Java CI - test workflow on main:

commit	result	run
83448ea6	success	job
7469a248	failure	job
9337abf1	failure	job
50130d37	failure	job
e64160a3	failure	job
64aaa29f	success	job
4bd617e1	failure	job
af0dc719	failure	job

6 of 8 failed. These are unrelated commits (a logger config change, an SQL arithmetic fix, a Duration fix, ...), so
this is the suite, not the changes.

The failures are non-deterministic, not a broken test

The clearest evidence: the same commit was run twice on #5685 and failed both times on disjoint sets of tests.

run	failing tests
1st (53m33s)	RaftMigrationCompactionRaceIT.migrationStyleBulkInsertAcrossManyTypesNeverEmitsCannotFindIndexes
2nd (55m55s)	Issue5410AbandonedTicketReleaseIT.abandonedEntryReleasesItsPhase2TicketOnceApplied, Issue5569SlotMergeDeleteRaftIT.mergedDeletesReplicateIntact, RaftHTTPGraphConcurrentIT.oneEdgePerTxMultiThreads

Every test that failed in run 1 passed in run 2, and vice versa. Final tally of run 2: Tests run: 199, Failures: 2, Errors: 1, Skipped: 4.

Shape of the failures

All timing-related rather than assertion-related. Representative:

com.arcadedb.server.ha.raft.ReplicationDispatchedTimeoutException:
  Group commit interrupted while awaiting quorum result (entry was dispatched to Raft; outcome unknown)

and the runs are full of leader churn:

WARNI [ArcadeStateMachine] Leader churn: ArcadeDB_2 (localhost:2482) re-elected (term=2, 7958 ms since last
leader change). Either a heartbeat stall triggered an election (CPU/GC pauses, disk stalls, network blips, or
appender threads saturated by bulk-load replication ...)

That is consistent with a 3-node Raft cluster not getting enough CPU/IO headroom on a shared GitHub runner: elections
fire, quorum waits time out, and whichever test happens to be mid-commit is the one that fails.

Why it matters

• At ~55 minutes it is the longest job in the workflow, so every red run costs a lot of wall-clock to re-run - and a
  re-run is likely to fail on something else.
• Because it is red on main too, it carries no signal: a genuine HA regression in a PR would be indistinguishable
  from the background noise. This came up concretely on fix(engine) #5677: store a LINK hash-index key compressed, and refuse an unencodable key type at creation #5685, where establishing that the failure was pre-existing
  took two 55-minute runs plus a survey of main's history.

Suggested directions

Rather than a specific fix, the useful first step is deciding which of these it is:

1. Runner capacity. If elections are firing because the 3 in-process nodes are starved, the durable fix is
   raising arcadedb.ha.electionTimeoutMin/Max (and possibly EXPLICIT_LOCK_TIMEOUT / COMMIT_LOCK_TIMEOUT, both
   5 s by default) for the CI profile only, so a real deadlock still fails fast locally. engine/CLAUDE.md already
   documents that these timeouts are a known source of intermittent CI failures under contention.
2. Genuine races. If the same handful of ITs dominate across many runs, they are worth fixing individually
   rather than blanket-retiming. Collecting per-test failure counts over ~20 runs would tell them apart.
3. Interim: consider marking the suite non-blocking, or adding a single automatic retry for it, so its result
   stops being ignored by default. Whatever is chosen, it should be explicit - the current state is an implicit
   "ignore this check" that everyone has to learn.

Related: #5701 (the coverage report does not wait for this job, so its coverage is silently missing from every PR).

Found while verifying that an HA failure on #5685 (issue #5677) was unrelated to that change - ha-raft/src
contains no reference to HashIndex, INDEX_TYPE.HASH or UNIQUE_HASH, so the changed code never executes in this
suite.

[lvca]

Corroborating data from #5772 (merged as 85eb71bb1), which sat through four CI runs over ~8 hours and hit this every time. Recording it because the "different test each run" pattern held exactly, and because one comparison rules out the PR as a cause with no ambiguity.

On the PR branch:

Run	Failing test(s)
1	Issue5410AbandonedTicketReleaseIT.abandonedEntryReleasesItsPhase2TicketOnceApplied + Issue5569SlotMergeDeleteRaftIT.mergedDeletesReplicateIntact
2	RaftServerDatabaseSqlScriptIT.executeSqlScript
3	same pair as run 1

On main, same window:

Run	Failing test(s)
30761436982	Issue5410AbandonedTicketReleaseIT + Issue5569SlotMergeDeleteRaftIT — the identical pair
30760991652	RaftReplicationChangeSchemaIT.schemaChangesReplicate
30759219297	Issue5569SlotMergeDeleteRaftIT + RaftPriorityRejoinIT.leaderRestartThenReplicaRestartConverges

Six runs, five distinct tests, main red on three consecutive runs. The identical-pair match between a PR run and a main run is the cleanest evidence I have that the job's redness is independent of what is being merged.

The failure modes are not one bug wearing different hats — they look like several symptoms of quorum/replication timing:

• Issue5569SlotMergeDeleteRaftIT: AssertionError: Update returned null: record=N seq=M (seen with record=7/seq=38 and record=2/seq=12) — a sqlscript UPDATE returning null under concurrent churn.
• Issue5410AbandonedTicketReleaseIT: Node 0 must hold both vertices, expected 2 but was 1 — the abandoned entry not converging within the wait.
• RaftServerDatabaseSqlScriptIT: ReplicationDispatchedTimeoutException: Group commit interrupted while awaiting quorum result, with a suppressed DatabaseAreNotIdentical: Types: DB1 6 <> DB2 7 from the end-of-test comparison.

Worth noting the practical cost, which is the reason I am adding this rather than just moving on: over four runs I had to individually classify every failure to establish the PR was not the cause, and on one run unit-tests (Issue4960LSMCloseDuringCompactionTest, failing in TestHelper.checkDatabaseIntegrity at teardown) and builder-tests (npm install) were red too and cleared on re-run with no code change. A permanently-red job trains everyone to skim past red CI, which is precisely when a real regression slips through.

[lvca]

Another data point from PR #5982 (unrelated change: SimpleDateFormat thread-safety fix in the integration module, nowhere near HA/Raft code). ha-integration-tests failed on 3 consecutive runs of the same commit (1 original + 2 manual reruns), each with a different secondary failure:

• Run 1 (job 93212666300): Issue5410AbandonedTicketReleaseIT.abandonedEntryReleasesItsPhase2TicketOnceApplied — 42.24s — Node 0 must hold both vertices: expected 2L but was 1L, plus a suppressed DatabaseComparator$DatabaseAreNotIdentical: Page PageId(graph/28/0) has different versions on databases. DB1 1 <> DB2 2
• Run 2 (job 93219255086): same Issue5410AbandonedTicketReleaseIT test failed again at 42.25s (near-identical to run 1), plus RaftRemoteReadYourWritesIT.readerCommitIndexAdvancesMonotonicallyAcrossWrites errored at 11.87s
• Run 3 (job 93225480100): same Issue5410AbandonedTicketReleaseIT test failed a third time at 42.77s, plus Issue5569SlotMergeDeleteRaftIT.mergedDeletesReplicateIntact errored at 14.80s

Issue5410AbandonedTicketReleaseIT failed in all 3/3 runs at nearly identical elapsed time (42.2-42.8s), which reads less like random scheduling noise and more like a deterministic timeout/threshold being hit consistently on these runners (GH Actions job log shows CPUs=4 MAXRAM=3.90GB per node, 4-node Raft cluster in-process — i.e. CPU/memory-constrained). Might be worth someone checking whether Issue5410AbandonedTicketReleaseIT specifically has a hardcoded wait/timeout around the ~40s mark that's too tight for these runner specs, since it reproduced identically three times in a row rather than intermittently.

No action taken here since it's out of scope for #5889/#5982 — just logging the data point since I had full logs pulled while investigating.

[lvca]

Another data point corroborating the ~42-45s deterministic-timeout hypothesis from the previous comment: PR #6019 (unrelated change: Type.convert() NaN guard, LSM close-during-compaction test fix, HA resync-retry test dedup - zero overlap with the Raft ticket-release code path), job 93284066536.

Issue5410AbandonedTicketReleaseIT.abandonedEntryReleasesItsPhase2TicketOnceApplied failed at 44.74s - [Node 0 must hold both vertices] expected: 2L but was: 1L, same suppressed DatabaseComparator$DatabaseAreNotIdentical: Page PageId(graph/28/0) has different versions on databases. DB1 1 <> DB2 2 shape as the three PR #5982 runs (42.2-42.8s). Issue5569SlotMergeDeleteRaftIT.mergedDeletesReplicateIntact also failed in the same run (Update returned null: record=5 seq=40) - same pairing noted in the very first comment on this issue.

That's now 4/4 observed elapsed times for Issue5410AbandonedTicketReleaseIT clustered in a 42.2-44.7s band rather than scattered across the run - adds weight to checking that test for a hardcoded wait/timeout in that range that's too tight for the CI runner's CPU/memory constraints, rather than treating it as pure scheduling noise.

[lvca]

Another confirmed sighting, from PR #6150 (merged) - a Cypher-only change (query/opencypher/**, zero overlap with HA/Raft/clustering code) that hit ha-integration-tests on 3 consecutive reruns, each time on a different disjoint set of tests:

rerun	failing tests
1	Issue5410AbandonedTicketReleaseIT.abandonedEntryReleasesItsPhase2TicketOnceApplied
2	Issue5569SlotMergeDeleteRaftIT + Issue5410AbandonedTicketReleaseIT (again)
3	RemoteStickyConnectionStrategyIT.pinIsReleasedAfterRollbackAndAllowsSubsequentStickyTransaction + RaftCommandReadConsistencyIT

4 distinct tests across 3 runs, no overlap with the actual diff - consistent with this issue's existing evidence that it's the suite/environment, not any specific change. Adding this as one more corroborating data point; each run took 30-90 minutes, so this remains expensive to work around per-PR.