Snapshot install advances the applied index without notifying waitForAppliedIndex(), causing spurious LINEARIZABLE 503s and READ_YOUR_WRITES timeouts

[ruispereira]

Summary

A follower that catches up through a snapshot install advances its applied index without waking the threads blocked in RaftHAServer.waitForAppliedIndex(). They keep sleeping until quorumTimeout even though the index they are waiting for has already been reached.

The visible effects:

• LINEARIZABLE reads fail with ReplicationException -> HTTP 503, although the local state machine was in fact up to date.
• READ_YOUR_WRITES reads pay the full quorumTimeout and then log consistency degraded to EVENTUAL, a warning that misreports what happened.

Version: 26.8.1.

Detail

The wait is a plain wait/notify on applyNotifier:

// ha-raft/src/main/java/com/arcadedb/server/ha/raft/RaftHAServer.java:2121
public void waitForAppliedIndex(final long targetIndex, final boolean throwOnTimeout) {
  if (targetIndex <= 0)
    return;
  try {
    final long deadline = System.currentTimeMillis() + quorumTimeout;
    synchronized (applyNotifier) {
      while (getLastAppliedIndex() < targetIndex) {
        final long remaining = deadline - System.currentTimeMillis();
        if (remaining <= 0) {
          ...
        }
        applyNotifier.wait(remaining);
      }
    }

notifyApplied() has exactly one caller in the codebase, at the end of ArcadeStateMachine.applyTransaction():

// ha-raft/src/main/java/com/arcadedb/server/ha/raft/ArcadeStateMachine.java:636
// Wake up any threads waiting for this index (READ_YOUR_WRITES, waitForLocalApply)
final RaftHAServer raftHA = this.raftHAServer;
if (raftHA != null) {
  raftHA.notifyApplied();

But applyTransaction() is not the only path that advances the applied index. reinitialize() — called by Ratis's StateMachineUpdater.reload() after a snapshot install, as the javadoc in that same file describes — seeds it directly and does not notify:

// ha-raft/src/main/java/com/arcadedb/server/ha/raft/ArcadeStateMachine.java:438
lastAppliedIndex.set(snapshotIndex);
...
updateLastAppliedTermIndex(snapshotInfo.getTerm(), snapshotIndex);

There is no notifyApplied() anywhere in reinitialize().

The waiter's loop condition reads RaftHAServer.getLastAppliedIndex(), which comes from Ratis itself:

// RaftHAServer.java:2191
return raftServer.getDivision(raftGroup.getGroupId()).getInfo().getLastAppliedIndex();

so the condition does become true after the snapshot is installed. The waiter simply is never told, and wait(remaining) only returns at the deadline.

Failure scenario

1. A follower falls far enough behind that the leader ships a snapshot instead of log entries.
2. A client read arrives at that follower with X-ArcadeDB-Read-After: N (READ_YOUR_WRITES) or with LINEARIZABLE.
3. The handler calls waitForAppliedIndex(N); the local applied index is still below N, so the thread waits.
4. The snapshot install completes and reinitialize() sets the applied index past N. No notification is sent.
5. The waiter sleeps for the remainder of quorumTimeout.
6. LINEARIZABLE then throws ReplicationException and the read fails with 503; READ_YOUR_WRITES logs a degradation warning and serves the read.

The window closes as soon as any subsequent entry is applied, since that calls notifyApplied() and the loop re-checks. So the bug bites hardest on a cluster that is quiet right after a catch-up — which is exactly when a snapshot install happens and when these waits are most likely to be pending.

Suggested fix

Notify at the end of reinitialize() whenever the snapshot seeded a new index:

lastAppliedIndex.set(snapshotIndex);
updateLastAppliedTermIndex(snapshotInfo.getTerm(), snapshotIndex);

final RaftHAServer raftHA = this.raftHAServer;
if (raftHA != null)
  raftHA.notifyApplied();

More robustly, since the waiter's condition is sourced from Ratis rather than from the state machine's own field, the two can drift apart for any Ratis-internal apply (configuration entries, leader no-op entries) that never reaches applyTransaction. A bounded re-check — applyNotifier.wait(Math.min(remaining, someSmallInterval)) — would make the wait robust against every missed-notification path rather than just this one, at the cost of a periodic wake-up.

[ruispereira]

Re-checking the whole file for the fix's sake: notifyApplied() has exactly one call site (ArcadeStateMachine.java:638, inside applyTransaction), while three paths advance lastAppliedIndex:

line	method	notifies?
622	applyTransaction	yes (638)
438	reinitialize()	no — the one this issue reports
1178	notifyInstallSnapshotFromLeader(...)	no

So there is a second unnotified path: the follower snapshot-install from the leader. It sets lastAppliedIndex.set(snapshotIndex) at line 1178 and returns at 1185 without ever calling notifyApplied(), and it does not route through reinitialize() (verified — the method contains no reinitialize call).

A fix that only adds the notify to reinitialize() would still leave a thread blocked in waitForAppliedIndex() stuck through a leader-driven snapshot install. Both sites — 438 and 1178 — need the notify, which is the argument for the more robust option from the issue body: a bounded re-check in waitForAppliedIndex() (applyNotifier.wait(min(remaining, interval))) covers every advance path, including any future one, rather than chasing call sites.

[ruispereira]

The waiter side of the matrix, to complete the picture: applyNotifier.wait(...) has two callers, not one.

waiter	line
waitForAppliedIndex(targetIndex, ...)	RaftHAServer.java:2143
waitForLocalApply()	RaftHAServer.java:2179

Both block on applyNotifier and both are woken only by the single notifyApplied() at ArcadeStateMachine.java:638. Crossed with the two advance paths that never call it (reinitialize() at 438, notifyInstallSnapshotFromLeader() at 1178), that is 2 waiters × 2 silent advance paths = 4 stuck combinations, not one.

waitForLocalApply() backs linearizable follower reads, so the same snapshot-install window that stalls waitForAppliedIndex also stalls those.

This is the decisive argument for fixing the wait rather than the call sites: patching notify into both advance paths works today but is one future lastAppliedIndex.set(...) away from regressing, whereas a bounded re-check — applyNotifier.wait(Math.min(remaining, interval)) — in both waiters is correct against every advance path, present and future.