Skip to content

id(list/peers): hand-rolled clients still cap unbounded List/ListPeers responses at 1 MiB — a ~15–25k-item store cannot be listed at all, and #1617's shared read_capped_response was never applied to them #1761

Closed

Description

@dezren39

Problem

id list fails outright against any node whose store is large enough that the List response passes 1 MiB — the same defect class #1617 fixed for remote find, left behind on every other hand-rolled unbounded-response read.

  • The server's List handler collects every item, unbounded: pkgs/id/src/protocol.rs:554-565 builds the full Vec<(Hash, String)> and the protocol writer sends it whole.
  • Four hand-rolled clients read that response with a literal recv.read_to_end(1024 * 1024):
    • src/commands/list.rs:81 — id list via the local serve instance
    • src/commands/list.rs:147 — id list <remote-node>
    • src/commands/repl.rs:430 — REPL list in connected modes
    • src/commands/repl.rs:839 — REPL list_on_node (@NODE_ID)
  • read_to_end(max) errors when the stream exceeds the cap, and these sites propagate it with bare ?, so the failure surfaces as a raw "stream too long" transport error — not a size message.
  • id(find): remote find fails outright past ~1k matches — server returns all matches unbounded, both clients cap the response at 64 KiB #1617's fix (merged today in fix(id): remote find reads through the shared 64 MiB response cap (#1617) #1719, main fb3d6372) added the shared helper for exactly this: src/meta_client.rs:63-75 — read_capped_response reads through MAX_RESPONSE_BYTES (64 MiB) and capped_response_error names the limit on overflow. Its doc comment says hand-rolled paths use it "instead of a per-call-site literal bound, so every response read shares the one named cap and cannot drift back to a smaller size" — but only the two find sites were migrated. List (and peers, below) still carry the literal bound the helper exists to eliminate.
  • The same literal 1 MiB bound sits on the unbounded ListPeers response (handler src/protocol.rs:643-649; each PeerAnnouncement is ~51 wire bytes): src/commands/peers.rs:215, src/commands/peers.rs:255, src/commands/repl.rs:1032, src/commands/repl.rs:1083, and the serve-side stale-peer probe src/commands/serve.rs:1329 (there an over-cap response reads as a dead peer and the still-live peer is removed as stale).

Fixed-size responses (Get/Delete at 64 KiB in get.rs/repl.rs) are not in scope — those responses are one hash / one bool and cannot grow with the store.

Evidence (verified this run on main fb3d6372)

Postcard sizing probe on the laptop (~/git/fleet/probe-2026-10-28-laneB-listcap/, wire-shape replicas of MetaResponse::List / ListPeers exactly as in the #1617 find probe; log in the goal's hidden_files/probe-2026-10-28-laneB-listcap/probe.log):

  • List item wire size: ~41 bytes (9-byte name), ~52 bytes (20-byte name), ~72 bytes (40-byte name)
  • First item count whose encoded List response exceeds 1 MiB: 24,967 items (9-byte names), 19,785 (20-byte), 14,365 (40-byte)
  • A 25,000-item list with 20-byte names encodes to 1,325,004 bytes — over the cap
  • ListPeers crosses 1 MiB at ~20,165 announcements (~51 bytes each)

So a node with roughly 15–25k stored blobs — unremarkable for a long-lived store — cannot be listed at all from the CLI or the REPL, locally-served or remote, while the direct-store path (no serve running) iterates the store and works, and remote find on the same node now works to 64 MiB. The inconsistency sends debugging toward connectivity, exactly as #1617 did.

Why it matters

list is the inventory verb: if a node's store outgrows ~20k entries, its contents become unlistable through every networked path, with a misleading transport error. The shared-cap helper and its regression-test pattern already exist from #1617/#1719; leaving four (plus five peers) call sites on the literal bound is the drift that fix's own doc comment warns against.

Acceptance criteria

  • All four List response reads (commands/list.rs ×2, commands/repl.rs ×2) go through meta_client::read_capped_response (or the unified MetaClient), so a List response up to MAX_RESPONSE_BYTES succeeds at every site.
  • The five ListPeers reads (commands/peers.rs ×2, commands/repl.rs ×2, commands/serve.rs probe) use the same shared helper; in the serve probe, an over-cap response is not silently indistinguishable from a dead peer (at minimum it follows the same named-error path before the stale decision, or the distinction is logged).
  • Exceeding the shared cap on any of these paths produces the named "response too large: exceeds the 67108864-byte limit" error from capped_response_error, not a bare "stream too long".
  • A regression test in the shape of meta_client.rs's find_response_larger_than_64_kib_reads_through pins a List response above 1 MiB reading through intact.
  • No remaining literal read_to_end(1024 * 1024) response reads in pkgs/id/src/commands/ or commands/serve.rs (fixed-size Get/Delete 64 KiB reads are explicitly out of scope and may stay).
  • Local direct-store list behaviour, ordering, and output format unchanged; existing id crate tests still pass (laptop or CI).

Dedup

Searched open + closed issues for list/response-cap/read_to_end terms: only #1617 (closed, fixed by #1719) — it was scoped to find's 64 KiB bound and cited List's 1 MiB as the sibling with "far more headroom"; no filing covers the List/ListPeers sites hitting their own bound, and #1719 migrated only the find sites.

Activity

  1. developing-today commented on Oct 28, 2026

    @developing-today
    Contributor
  2. added
    queue/queuedHas a queued task in the Supabase agent queue
    source/creatorFiled by the new-issue-creator job
    task/triageQueue packet kind: verify, reproduce, and route this item
    priority/normalOrdinary priority (the default)
    app/idPackage/app: id (pkgs/ or apps/ in this repo)
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    effort/sSmall: expected under 2 hours of work
    on Oct 28, 2026
  3. dezren39 commented on Oct 28, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · filed

  4. added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed on Nov 1, 2026
  5. 34 remaining items

  6. added
    status/parkedDeliberately paused with a recorded resume condition
    and removed
    queue/claimedQueue worker lifecycle state
    on Nov 1, 2026
  7. dezren39 commented on Nov 1, 2026

    @dezren39
    MemberAuthor

    agent-note:v1 — parked (queue task 4800, Lane A)

    PR #3277 head unchanged (8e6aad15d); Codex has no findings on the PR (zero Codex reviews/comments). CI is still stuck in the fleet runner backlog: the id run 41260470886 has been queued since 01:17Z (~75 min at re-check; 27 checks queued, 1 in progress, 2 success). Parked another 20m with the same resume condition: checks green → merge → close this issue → complete task 4800 as fixed_merged.

  8. added
    queue/claimedQueue worker lifecycle state
    and removed
    status/parkedDeliberately paused with a recorded resume condition
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    on Nov 1, 2026
  9. dezren39 commented on Nov 1, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · picked up

  10. added a commit that references this issue on Nov 1, 2026
  11. dezren39 commented on Nov 1, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

app/idPackage/app: id (pkgs/ or apps/ in this repo)effort/sSmall: expected under 2 hours of workkind/bugSomething isn't working (prefixed form of the classic bug label)priority/normalOrdinary priority (the default)source/creatorFiled by the new-issue-creator jobtask/triageQueue packet kind: verify, reproduce, and route this item

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions