Repository navigation
id(waiting): WaitingPool's arrived set grows with every event ever applied — +37 MB per 2M arrivals in a pool that reports itself empty, and no production caller ever reads it #1730
Description
Problem
WaitingPool (pkgs/id/src/waiting.rs) remembers every key that has ever arrived, forever:
arrived: HashSet<K>(waiting.rs:135) — "Keys that have arrived, and stay arrived."WaitingPool::arrived(waiting.rs:324-344) inserts the key into that set on every call and nothing ever removes from it: there is no eviction, no expiry (the age cap applies only to held events), and no reset short of dropping the pool.- The event store holds one pool for the life of the store (
event_store/mod.rs:342, built at :382) and callspool.arrivedon every applied event and every fork-proof header, viawaiting_rows.rsrelease(:262) andpresent_proof(:274). So a long-lived node's pool grows by one hash set entry per event it has ever applied — while the pool's actual held set is capped at 256 per author.
The growth buys nothing in production: the store's hold path uses pool.restore (waiting_rows.rs hold), which consults only by_key, never arrived. The set is read only by the pool's own insert/park duplicate-and-dependency filter (waiting.rs:202, :207, :234), which no production caller uses — the store is the only non-test constructor of a pool besides the property tests. In the running server the set is effectively write-only memory.
Evidence (verified this run, main 8c6bdc09)
- Line trace of waiting.rs:
arrivedset is inserted at :343; searches for removal/clear of the set across the file and its callers find none. - Caller trace:
grepforpool.insert/pool.park/.insert(on the pool acrosspkgs/id/src/event_store/finds no production use; holds go throughrestore, releases througharrived. - Laptop probe executed this run (probe crate against the real
idcrate as a path dependency,~/git/fleet/probe-2026-10-28-lanea-waiting/on the laptop;waiting.rssha2569e693d746d…identical on sandbox main8c6bdc09and the laptop; log inhidden_files/probe-2026-10-28-laneA-waiting/probe.log): 2,000,000arrived()calls with distinct keys and no waiters moved RSS 2,816 KB → 39,816 KB (+37 MB) while the pool reportslen() == 0andis_empty() == true; the same key arriving 2,000,000 times moved RSS by 4 KB. Behavioural check in the same probe: re-inserting an already-arrived key is silently swallowed (ready=0, held=0), confirming the set — not any held state — is what retains the keys.
Distinct from #1567 (no total cap on held waiting events — that is about events still waiting; this is about keys of events already applied, which no cap touches) and from #1726 (outbox retention — different store).
Why it matters
id serve is a long-lived process syncing every event of every world it hosts. An uncapped, never-read set that grows per applied event is a slow memory leak proportional to total history, not to live state — the opposite of the pool's own design, where held events are capped per author and aged out in 7 days.
Acceptance criteria
- The fix direction is pinned in triage: e.g. drop the
arrivedset from the store's path (the store already knows applied keys from its tables), bound it, or makerestore-based callers not populate it — while keeping the pool's standaloneinsert/parkcontract correct for direct users, or explicitly changing that contract with its docs. - After the fix, applying N events through the store leaves the pool's memory independent of N (held events aside); a regression test or property test demonstrates no per-applied-event growth (the existing
tests/waiting_properties.rsharness is a natural home). - Duplicate/replay behaviour is unchanged: re-delivering an already-applied event is still recognised (by the store's applied table), and a waiting event whose dependency applies is still released.
- Existing waiting pool unit tests and
tests/waiting_properties.rspass (laptop or CI).
Activity
developing-today commentedon Oct 28, 2026
CODE-696
dezren39 commentedon Oct 28, 2026
Filed this run by new-issue-creator-code-a (lane A). Found via: line + caller trace of pkgs/id/src/waiting.rs and pkgs/id/src/event_store/ (store holds go through restore, which never reads the arrived set), plus an executed laptop probe against the real id crate (~/git/fleet/probe-2026-10-28-lanea-waiting/): 2M distinct arrived() keys = +37 MB RSS in an empty pool; same-key control = +4 KB. Verified on main 8c6bdc0 (waiting.rs sha256 9e693d746d, identical on laptop). Distinct from #1567 (held-event total cap). Enqueued as a repo-task/v1 triage packet in repo-health-code-pilot.
61 remaining items
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: parked
task: 7494 (repo-health-code-pilot; worker repo-worker-code; OpenChamber scheduled lane bf419306-c4ef-470f-a553-beabf93239de)
state: parked for 30 minutes; work/assigned retained, queue/parked + status/parked synchronized.
blocker: Fresh check confirms canonical PR #3752 remains OPEN at af024ce, authored and solely assigned to Drew (dezren39). No PR, branch, worktree, review, or merge changes were made by this lane.
resume: only if Drew releases ownership or PR #3752 closes/merges; then revalidate issue #1730, PR #3752, reviews, and checks fresh.
bookkeeping: openchamber scheduled lane — sandbox bookkeeping deferred to fleet reconciliation.
dezren39 commentedon Nov 4, 2026
agent-note:v1
{"kind":"handoff","task":7494,"lane":"code-v","execution_lane":"opencode","state":"parked","note":"task 7494 handed off to opencode claimant by lane code-v. Fresh revalidation: issue #1730 OPEN; canonical PR #3752 OPEN, authored and assigned to Drew (dezren39) — ownership hard stop, PR/branch/worktree untouched. Resume only if Drew releases ownership or PR #3752 closes/merges."}
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 5, 2026
agent-note:v1
event: picked_up
task: 7494 (repo-health-code-pilot; worker repo-worker-code; OpenChamber scheduled lane ec118cad-0fd3-4652-91c4-27432f2f2efd)
state: claimed; resuming from handed_off checkpoint.
Fresh issue state: #1730 remains OPEN and links canonical PR #3752, which is still OPEN and authored by dezren39 (Drew). Prior queue progress identifies it as Drew-owned and says not to touch its PR, branch, worktree, review, or merge. No work has been touched. I am checking current PR ownership/state before deciding the required terminal disposition.
dezren39 commentedon Nov 5, 2026
agent-note:v1
event: escalated
task: 7494 (repo-health-code-pilot; worker repo-worker-code; OpenChamber scheduled lane ec118cad-0fd3-4652-91c4-27432f2f2efd)
state: blocked (escalated-to-user)
Read attached: fresh GitHub issue #1730 is OPEN and links PR #3752, which is OPEN, authored by dezren39, assigned solely to dezren39, and directly fixes #1730. Its current head is queue/4796-1730-waiting-arrived at af024ce. Checks show SUCCESS, but ownership—not CI—is the blocker. This lane did not touch the PR, branch, worktree, review, or merge, and created no competing PR.
Decision needed: Drew must release ownership, or PR #3752 must close/merge, before task 7494 resumes. Shared queue/claimed, status/in-progress, and work/assigned labels remain because the canonical PR is still active; queue/blocked, status/blocked, and user/escalated record this task's disposition.