Repository navigation
id(outbox): finished entries are kept forever — the in-memory map, the append-only journal and the deliveries view all grow with the server's entire delivery history #1726
Description
Problem
The envelope outbox (pkgs/id/src/envelope_outbox.rs) never forgets a finished delivery:
Outbox.entries(envelope_outbox.rs:121) is aBTreeMap<String, Entry>holding every payload ever queued. The only removal anywhere in the module is theWithdrawnarm ofapply(envelope_outbox.rs:322), andwithdrawitself refuses any entry that has been attempted or finished — so aDelivered/Refused/GaveUpentry can never leave the map for the life of the process.- The on-disk journal has the same shape:
commit(envelope_outbox.rs:278) only appends one JSON line per queue/attempt/finish, andopen(envelope_outbox.rs:130) only ever truncates a torn final line (set_len, :160). No compaction or pruning exists anywhere in the crate (verified by searchingsrc/for the outbox file's writers:commit/openare the only ones). - The cost is paid on every restart and every directory view:
openreplays and re-parses every line ever written, anddeliveries()(pkgs/id/src/directory_view.rs:432) iterates all entries and clones aDeliveryViewfor each one the viewer may see. The view's own doc promises only envelopes "still in, or recently left, the outbox" (directory_view.rs:86) — the implementation has no recency bound at all, so a long-lived server's directory view grows linearly with its entire delivery history, forever.
This is a queue, not a journal of record: the directory log is where history lives. Nothing else in the crate reads finished outbox entries except deliveries() and entry() status lookups.
Evidence (verified this run, main 8c6bdc09)
- Line trace of
envelope_outbox.rs(whole file):applyarms areQueued,QueuedArtifact,Attempted,Finished,Withdrawn; onlyWithdrawnremoves, andwithdraw()(same file) errors unlessattempts == 0 && finish.is_none(). grepoverpkgs/id/srcfor compaction/pruning of the outbox (set_len,remove, rewrite): the only hits are the torn-tail truncation inopenand theWithdrawnremoval above.deliveries()atdirectory_view.rs:432-448filters by viewer only — no age, count, or finish-time bound — while its field doc at :86 says "recently left".
Distinct from #1620 (torn-tail acceptance in the same file) and from #1713 (SaveRateLimiter map growth in web/) — this is the outbox store itself, its journal file, and its directory view.
Why it matters
A world server runs for months and flushes its outbox on a timer (world_hub.rs spawn_outbox_flush). Every friend envelope and artifact push adds permanent in-memory entries (including full payload clones), permanent journal lines, restart replay time, and directory-view response size. Growth is slow but strictly unbounded and invisible until the file and the view are large.
Acceptance criteria
- A retention rule is pinned and documented at
Outbox(e.g. finished entries kept for a bounded age or bounded count, still long enough fordeliveries()to honestly show "recently left" ones). - Finished entries past the bound are dropped from the in-memory map, and the journal is compacted (rewritten without the dropped history) at a safe point, e.g. on
openafter replay. -
deliveries()output is bounded by the same rule; a test with an old finished entry and a recent finished entry shows only the recent one. - Existing outbox tests keep passing (
envelope_outbox+envelope_netflush tests), including torn-tail handling (id(outbox): a parseable torn tail is accepted, then merged into the next record — Outbox::open only drops unterminated final lines that fail to parse #1620's area must not regress: compaction must keep the crash-torn-tail rule working). - A restart test: after compaction, reopening the file yields the same due/pending set as before.
Metadata
Metadata
Assignees
Labels
Package/app: id (pkgs/ or apps/ in this repo)Small: expected under 2 hours of workPerformance problem or optimizationNice to have; do when convenientQueue or user escalation stateFiled by the new-issue-creator jobQueue or user escalation stateQueue packet kind: verify, reproduce, and route this itemQueue or user escalation state
Activity
developing-today commentedon Oct 28, 2026
CODE-694
dezren39 commentedon Oct 28, 2026
Filed this run by new-issue-creator-code-a (lane A). Found via: full line trace of pkgs/id/src/envelope_outbox.rs plus a crate-wide search for any outbox compaction/pruning (none exists outside the torn-tail truncation), and the deliveries() view in pkgs/id/src/directory_view.rs:432 vs its 'recently left' doc at :86. Verified on main 8c6bdc0. Enqueued as a repo-task/v1 triage packet in repo-health-code-pilot.
39 remaining items
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
agent-note:v1
state: picked-up
queue: repo-health-code-pilot
task: 8101 (triage, worker repo-worker-code, lane 262eb403-3347-4a7c-ba6b-e7bc263ca853)
by: openchamber-scheduled-lane
at: 2026-11-04T04:09Z
Resuming the prior handed-off triage. Fresh issue state is OPEN; PR #4710 (
queue/6512-1726-outbox-retention) is still open and directly linked. I’m checking the PR and current queue ownership before deciding whether this is already routed or needs a new packet.dezren39 commentedon Nov 4, 2026
agent-note:v1
state: escalated
queue: repo-health-code-pilot
task: 8101 (triage, worker repo-worker-code, lane 262eb403-3347-4a7c-ba6b-e7bc263ca853)
by: openchamber-scheduled-lane
at: 2026-11-04T04:14Z
Triage verdict: the original outbox-retention problem remains actionable, and the canonical fix is already in open PR #4710 (
queue/6512-1726-outbox-retention); it has one passing check. I will not open a competing PR or modify this PR because it is authored/owned by the repo owner. Fresh review state has one unresolved high Copilot finding: compaction usesfs::rename(src, dst)while the destination exists, which typically fails on Windows and may makeOutbox::open()fail. The owner needs to choose/approve the replacement strategy (platform-specific atomic replace versus a documented non-atomic fallback) on the existing PR. No duplicate fix packet was enqueued. This task is blocked/escalated to the owner pending that decision; after the PR finding is addressed, resume the canonical PR and re-check its CI/review state.