Repository navigation
id(waiting): WaitingPool::restore ignores the arrived and ready-queue duplicate guards — an already-applied or ready event is held again and handed out a second time #3911
Description
Summary
WaitingPool::restore (pkgs/id/crates/id-core/src/waiting.rs:312-326) applies a narrower duplicate guard than insert and park: it checks only by_key (already held). insert (waiting.rs:234, guard at :243-246) and park (waiting.rs:273, guard at :281-284) refuse a key that is already held, has arrived, or is sitting in the ready queue. restore checks none of the latter two, so a key the pool already knows is finished or ready-to-hand-out can be held a second time — and later handed out a second time. That is exactly the double hand-out class closed #2001 fixed for insert (which is why the ready_keys index exists, waiting.rs:170-176); the fix never reached restore.
restore is not a test-only path: the event store uses it for every hold. Mirror::hold (pkgs/id/src/event_store/waiting_rows.rs:241) calls pool.restore(...), Mirror::park (waiting_rows.rs:320) re-holds proof headers through it, and the reload-on-open loop (pkgs/id/src/event_store/mod.rs:1015) restores every waiting row through it.
Verified this run (laptop probe, current main 33aefa7)
Probe crate on the laptop (~/git/fleet/probe-2026-11-01-laneA-restoreguard/) #[path]-includes the real waiting.rs from the laptop clone at the same HEAD (md5 9654300d490a030f3f5513c79692e717, identical in both clones) and exercises WaitingPool<u32, &str>:
- Case A — already-arrived key:
arrived(10)thenrestore(author, 10, [99], ev)→ the event is held (len=1,contains(10)=true); a laterarrived(99)releases it, handing an already-applied event to the caller a second time. Control:insertof the same arrived key is ignored (len=0). - Case B — ready-queue key (id(waiting): WaitingPool::insert's duplicate guard misses the ready queue — the same key inserted twice before take_ready is handed out twice #2001 class):
insert(author, 1, [], ev)(goes to the ready queue) thenrestore(author, 1, [99], ev2)→ the same key is also held;arrived(99)releases the held copy andtake_ready()returns the ready copy — the same key is handed out twice. Control:parkof a ready key is ignored (len(held)=0).
Why it matters
The pool's contract is that a key is in exactly one state — held, ready, or arrived — and is handed out at most once. restore silently breaks that invariant for two of the three states:
- An already-arrived (applied) event re-held via
restoreoccupies per-author/total cap slots until its dependency happens to arrive or the 7-day age cap drops it, and if the dependency does arrive it is released and applied again. - A ready key re-held via
restorecan be handed out twice (once from the ready queue, once on release), the precise failure id(waiting): WaitingPool::insert's duplicate guard misses the ready queue — the same key inserted twice before take_ready is handed out twice #2001 closed forinsert. - At the store layer,
Mirror::hold's Held/Skipped decision keys offpool.contains(&id)afterrestore(waiting_rows.rs:242-262), so the phantom hold also writes/keeps a waiting row for an event that is not waiting.
Design constraint — one caller does this on purpose
Mirror::present_proof (waiting_rows.rs:288-301) marks a fork-proof header arrived via pool.arrived(...) and then deliberately re-holds it through restore (Mirror::park, waiting_rows.rs:313-323): a proof header never applies, so it is parked for good to keep its conflicts seen. A blind copy of insert's arrived-guard into restore would silently break that park (the header would never be re-held). The fix therefore has to separate the intents — e.g. restore guards the ready queue unconditionally (no caller parks a ready key on purpose) and the arrived case gets an explicit held-for-good route for proofs — rather than one shared guard pasted over all three entry points. This is also why this issue is filed-only, not a trivial self-solve.
Acceptance criteria
-
WaitingPool::restorecan no longer hold a key that is in the ready queue, and a non-proof event whose key has already arrived is not re-held/re-released through theMirror::holdpath; the chosen guard shape is documented onrestorelike the other two entry points. - The fork-proof park still works:
present_proof/park_proofkeep a proof header held for good after its arrival, with a test pinning that behaviour (existing proof-header tests stay green). - Regression tests at the pool level, alongside
test_restore_keeps_the_cap_and_ignores_a_key_already_held(waiting.rs:728): restore-after-arrived holds nothing and releases nothing when its deps later arrive; restore-of-a-ready-key holds nothing and the key is handed out exactly once. - The reload-on-open and
Mirror::hold/parkpaths behave as before for genuinely waiting/parked events (existing waiting + waiting_rows tests stay green).
Dedup
Open/closed searches for "restore WaitingPool duplicate" and "waiting restore" (2026-11-01) return only #2001 (insert's ready-queue guard — closed, restore not covered), #1567 (total cap — closed), and #1730 (arrived-set growth — open, different defect). No issue covers restore's guard.
Activity
dezren39 commentedon Nov 1, 2026
🤖 Agent note · filed
WaitingPool::restore(pkgs/id/crates/id-core/src/waiting.rs:312) guards only already-held keys, unlikeinsert/parkwhich also refuse arrived and ready-queue keys — an already-applied event is re-held and released a second time, and a ready key is handed out twice (the id(waiting): WaitingPool::insert's duplicate guard misses the ready queue — the same key inserted twice before take_ready is handed out twice #2001 class). Verified with a laptop probe#[path]-including the real waiting.rs @33aefa7f (~/git/fleet/probe-2026-11-01-laneA-restoreguard/): restore-of-arrived held + released on its dep's arrival, restore-of-ready held + both copies handed out; insert/park controls ignored both. Note the design constraint in the issue:present_proofparks proof headers throughrestoreon purpose, so the fix must separate intents — filed-only, not self-solved.dezren39 commentedon Nov 1, 2026
🤖 Agent note · adopted as Phase 3a of design #4000
source/design-architect. Existing queued triage packet stands.109 remaining items
dezren39 commentedon Nov 4, 2026
🤖 Agent note · PR opened
dezren39 commentedon Nov 4, 2026
🤖 Agent note · parked
dezren39 commentedon Nov 4, 2026
🤖 Agent note · parked verification update
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
Merge pull request #5516 from developing-today/queue/6238-3911-waitin…