Description
Problem
id list fails outright against any node whose store is large enough that the List response passes 1 MiB — the same defect class #1617 fixed for remote find, left behind on every other hand-rolled unbounded-response read.
- The server's
Listhandler collects every item, unbounded:pkgs/id/src/protocol.rs:554-565builds the fullVec<(Hash, String)>and the protocol writer sends it whole. - Four hand-rolled clients read that response with a literal
recv.read_to_end(1024 * 1024):src/commands/list.rs:81—id listvia the local serve instancesrc/commands/list.rs:147—id list <remote-node>src/commands/repl.rs:430— REPLlistin connected modessrc/commands/repl.rs:839— REPLlist_on_node(@NODE_ID)
read_to_end(max)errors when the stream exceeds the cap, and these sites propagate it with bare?, so the failure surfaces as a raw "stream too long" transport error — not a size message.- id(find): remote find fails outright past ~1k matches — server returns all matches unbounded, both clients cap the response at 64 KiB #1617's fix (merged today in fix(id): remote find reads through the shared 64 MiB response cap (#1617) #1719, main
fb3d6372) added the shared helper for exactly this:src/meta_client.rs:63-75—read_capped_responsereads throughMAX_RESPONSE_BYTES(64 MiB) andcapped_response_errornames the limit on overflow. Its doc comment says hand-rolled paths use it "instead of a per-call-site literal bound, so every response read shares the one named cap and cannot drift back to a smaller size" — but only the two find sites were migrated. List (and peers, below) still carry the literal bound the helper exists to eliminate. - The same literal 1 MiB bound sits on the unbounded
ListPeersresponse (handlersrc/protocol.rs:643-649; eachPeerAnnouncementis ~51 wire bytes):src/commands/peers.rs:215,src/commands/peers.rs:255,src/commands/repl.rs:1032,src/commands/repl.rs:1083, and the serve-side stale-peer probesrc/commands/serve.rs:1329(there an over-cap response reads as a dead peer and the still-live peer is removed as stale).
Fixed-size responses (Get/Delete at 64 KiB in get.rs/repl.rs) are not in scope — those responses are one hash / one bool and cannot grow with the store.
Evidence (verified this run on main fb3d6372)
Postcard sizing probe on the laptop (~/git/fleet/probe-2026-10-28-laneB-listcap/, wire-shape replicas of MetaResponse::List / ListPeers exactly as in the #1617 find probe; log in the goal's hidden_files/probe-2026-10-28-laneB-listcap/probe.log):
- List item wire size: ~41 bytes (9-byte name), ~52 bytes (20-byte name), ~72 bytes (40-byte name)
- First item count whose encoded
Listresponse exceeds 1 MiB: 24,967 items (9-byte names), 19,785 (20-byte), 14,365 (40-byte) - A 25,000-item list with 20-byte names encodes to 1,325,004 bytes — over the cap
- ListPeers crosses 1 MiB at ~20,165 announcements (~51 bytes each)
So a node with roughly 15–25k stored blobs — unremarkable for a long-lived store — cannot be listed at all from the CLI or the REPL, locally-served or remote, while the direct-store path (no serve running) iterates the store and works, and remote find on the same node now works to 64 MiB. The inconsistency sends debugging toward connectivity, exactly as #1617 did.
Why it matters
list is the inventory verb: if a node's store outgrows ~20k entries, its contents become unlistable through every networked path, with a misleading transport error. The shared-cap helper and its regression-test pattern already exist from #1617/#1719; leaving four (plus five peers) call sites on the literal bound is the drift that fix's own doc comment warns against.
Acceptance criteria
- All four
Listresponse reads (commands/list.rs×2,commands/repl.rs×2) go throughmeta_client::read_capped_response(or the unifiedMetaClient), so aListresponse up toMAX_RESPONSE_BYTESsucceeds at every site. - The five
ListPeersreads (commands/peers.rs×2,commands/repl.rs×2,commands/serve.rsprobe) use the same shared helper; in the serve probe, an over-cap response is not silently indistinguishable from a dead peer (at minimum it follows the same named-error path before the stale decision, or the distinction is logged). - Exceeding the shared cap on any of these paths produces the named "response too large: exceeds the 67108864-byte limit" error from
capped_response_error, not a bare "stream too long". - A regression test in the shape of
meta_client.rs'sfind_response_larger_than_64_kib_reads_throughpins aListresponse above 1 MiB reading through intact. - No remaining literal
read_to_end(1024 * 1024)response reads inpkgs/id/src/commands/orcommands/serve.rs(fixed-sizeGet/Delete64 KiB reads are explicitly out of scope and may stay). - Local direct-store list behaviour, ordering, and output format unchanged; existing id crate tests still pass (laptop or CI).
Dedup
Searched open + closed issues for list/response-cap/read_to_end terms: only #1617 (closed, fixed by #1719) — it was scoped to find's 64 KiB bound and cited List's 1 MiB as the sibling with "far more headroom"; no filing covers the List/ListPeers sites hitting their own bound, and #1719 migrated only the find sites.
Activity
developing-today commentedon Oct 28, 2026
CODE-714
dezren39 commentedon Oct 28, 2026
🤖 Agent note · filed
34 remaining items
dezren39 commentedon Nov 1, 2026
agent-note:v1 — parked (queue task 4800, Lane A)
PR #3277 head unchanged (
8e6aad15d); Codex has no findings on the PR (zero Codex reviews/comments). CI is still stuck in the fleet runner backlog: theidrun 41260470886 has been queued since 01:17Z (~75 min at re-check; 27 checks queued, 1 in progress, 2 success). Parked another 20m with the same resume condition: checks green → merge → close this issue → complete task 4800 as fixed_merged.dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
awaiting_cicheckpoint: PR fix(id): migrate List/ListPeers 1 MiB response reads to read_capped_response (#1761) #3277 @8e6aad15d — verify checks + Codex findings fresh, merge if green, close id(list/peers): hand-rolled clients still cap unbounded List/ListPeers responses at 1 MiB — a ~15–25k-item store cannot be listed at all, and #1617's shared read_capped_response was never applied to them #1761, complete the task.queue/4800-1761-capped-responseMerge pull request #3277 from developing-today/queue/4800-1761-capped…
dezren39 commentedon Nov 1, 2026
🤖 Agent note · done
awaiting_cicheckpoint)read_capped_response(64 MiB); regression testlist_response_larger_than_1_mib_reads_through(30,000 items ≈1.5 MiB) passes on the laptop (pkgs/idcargo test, meta_client suite exit 0). Codex posted no findings on the PR; CodeRabbit passed — its one minor (pre-existingmax_peersseed/merge gap incmd_peers) was verified real but out of scope here and is already tracked as id(peers): --max-peers is not a hard cap — the crawl inserts a whole ListPeers response unchecked, and the RPC/gossip paths never consult it at all #1765; disposition replied on that thread. TheidGitHub workflow run for the head SHA sat queued ~100 min (runner backlog, infra), so verification ran on the laptop per the CI-first fallback; the merge follows the standing merge directive (verified change, PRs do not wait on queued CI).