Summary
Found during review of #6110 (fix for #5846). Not a regression from that PR - the underlying behavior
predates it - but #6110's fix (adding an explicit notifyApplied() call) makes the window tighter for
an already-blocked waiter, so filing separately rather than folding a broader fix into that PR.
Detail
In ArcadeStateMachine.reinitialize():
// ha-raft/src/main/java/com/arcadedb/server/ha/raft/ArcadeStateMachine.java
if (persistedApplied >= 0 && snapshotIndex > persistedApplied + snapshotGapTolerance) {
LogManager.instance().log(this, Level.INFO,
"Snapshot index %d is ahead of persisted applied index %d, will download from leader when available",
snapshotIndex, persistedApplied);
needsSnapshotDownload.set(true);
// ... watchdog scheduled to trigger an async download from the leader ...
}
lastAppliedIndex.set(snapshotIndex); // <-- unconditional, even when needsSnapshotDownload just became true
updateLastAppliedTermIndex(snapshotInfo.getTerm(), snapshotIndex);
raftHA.notifyApplied(); // <-- added by #6110When the gap check decides the on-disk snapshot is stale relative to the leader
(snapshotIndex > persistedApplied + snapshotGapTolerance), it flags needsSnapshotDownload and
schedules an async re-download/reconciliation - but the very next lines advance
lastAppliedIndex to the (possibly stale) snapshotIndex anyway and now also notify.
A LINEARIZABLE or READ_YOUR_WRITES read waiting on a target index <= snapshotIndex at that
moment can now wake and proceed immediately, potentially serving data from the stale local snapshot
before the flagged re-download/reconciliation has actually happened.
This is not new: a waiter that started blocking after lastAppliedIndex.set(snapshotIndex) already
saw the condition satisfied and returned without waiting at all, pre-#6110. What #6110 changes is that
an already-blocked waiter (one that started waiting before this line ran) now also wakes immediately
via the explicit notify, instead of only re-checking on its next scheduled poll/timeout - narrowing the
timing difference between "arrived before" and "arrived after" this line, and generally making the
already-possible early wake more likely to be observed.
Suggested follow-up
Worth deciding intentionally whether lastAppliedIndex (and now the notify) should be deferred until
after needsSnapshotDownload resolves (the watchdog-triggered or leader-change-triggered download
completes), rather than being set optimistically from the on-disk marker while it's already flagged as
possibly stale. That likely touches the PAUSED->RUNNING lifecycle transition semantics
StateMachineUpdater.reload() depends on, so it needs its own design pass rather than a quick patch.
References
Summary
Found during review of #6110 (fix for #5846). Not a regression from that PR - the underlying behavior
predates it - but #6110's fix (adding an explicit
notifyApplied()call) makes the window tighter foran already-blocked waiter, so filing separately rather than folding a broader fix into that PR.
Detail
In
ArcadeStateMachine.reinitialize():When the gap check decides the on-disk snapshot is stale relative to the leader
(
snapshotIndex > persistedApplied + snapshotGapTolerance), it flagsneedsSnapshotDownloadandschedules an async re-download/reconciliation - but the very next lines advance
lastAppliedIndexto the (possibly stale)snapshotIndexanyway and now also notify.A
LINEARIZABLEorREAD_YOUR_WRITESread waiting on a target index<= snapshotIndexat thatmoment can now wake and proceed immediately, potentially serving data from the stale local snapshot
before the flagged re-download/reconciliation has actually happened.
This is not new: a waiter that started blocking after
lastAppliedIndex.set(snapshotIndex)alreadysaw the condition satisfied and returned without waiting at all, pre-#6110. What #6110 changes is that
an already-blocked waiter (one that started waiting before this line ran) now also wakes immediately
via the explicit notify, instead of only re-checking on its next scheduled poll/timeout - narrowing the
timing difference between "arrived before" and "arrived after" this line, and generally making the
already-possible early wake more likely to be observed.
Suggested follow-up
Worth deciding intentionally whether
lastAppliedIndex(and now the notify) should be deferred untilafter
needsSnapshotDownloadresolves (the watchdog-triggered or leader-change-triggered downloadcompletes), rather than being set optimistically from the on-disk marker while it's already flagged as
possibly stale. That likely touches the PAUSED->RUNNING lifecycle transition semantics
StateMachineUpdater.reload()depends on, so it needs its own design pass rather than a quick patch.References