Repository navigation
id: PoC — multi-provider chunk-range retrieval in the get path (swarm scheduler over iroh-blobs) #2261
Description
Parent: #2050 (investigation: BitTorrent-style chunk retrieval for graph objects), epic #2056. Follow-up B of the #2050 investigation. Depends on the holder-discovery follow-up A (provider table) — a hand-fed provider list is acceptable for the PoC's benchmarks.
Problem
Retrieval in pkgs/id is single-source: src/commands/get.rs does one store.remote().fetch(conn, hash) over one connection. iroh-blobs (0.103, in Cargo.toml) already supports the swarm-shaped pieces — (hash, ChunkRanges) requests per provider, bao verified streaming (each 16 KiB chunk group verified against the blob's BLAKE3 root hash as it arrives, from any source, in any order), partial-blob state in FsStore, and providers reporting the exact missing location when they close a stream for data they lack. What does not exist is the scheduling policy that uses several holders at once.
Scope — PROOF OF CONCEPT (review-gated; do not build the permanent feature on it)
Implement the #2050 §5 retrieval flow as a PoC in the get path:
- Wanted-set = chunk-group ranges of the hash; assign ranges rarest-first across the provider set; disjoint ranges per holder; start with ≤4 concurrent sources.
- Verified groups persist to FsStore immediately (resume + become servable partial state). Bad group → discard, requeue range, deprioritise that holder for the hash. Missing-range stream close → shrink that holder's ranges, requeue elsewhere.
- Endgame: duplicate-request trailing in-flight ranges; first verified copy wins.
- Works in both discovery modes from id: investigate: BitTorrent-style chunk retrieval for graph objects #2050: observe-informed (held ranges known) and optimistic (observe off/refused — learn from missing-range reports).
- Nobody-home case surfaces distinctly ("0 holders for ranges X–Y", resumable from persisted partial state), never as a generic transfer failure.
Benchmarks required (n0 notes multiprovider single-blob download efficiency is still maturing upstream — measure, don't assume)
- 1 vs 2 vs 4 holders, complete holders; a partial-holder mix (e.g. two 50% holders with disjoint ranges); one holder leaving mid-transfer; observe-off mode.
- Report wall time + bytes duplicated vs the single-source baseline on the same blob sizes.
Acceptance criteria
- PoC branch/PR demonstrates a zero-local-copy fetch of a blob from ≥2 holders with per-range provenance logged.
- Benchmark table posted on this issue; failure/stall cases above behave as specified.
- The PR is explicitly a PoC: a review round decides what (if anything) becomes the permanent retrieval path. Nothing merges to main as "the feature" without that review (program rule from id: investigate: BitTorrent-style chunk retrieval for graph objects #2050 / Drew 2026-10-28).
Activity
developing-today commentedon Oct 29, 2026
CODE-1124
dezren39 commentedon Nov 1, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 1, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 1, 2026
🤖 Agent note · parked
pkgs/id/src/commands/get.rsis still single-source (verified at mainf1ddfb3d);iroh-blobs = "0.103"is inpkgs/id/Cargo.toml:56; follow-up A id: holder discovery for blob hashes — provider table, gossip announcements, wire the blob gate #2260 (holder discovery / provider table) is OPEN and actively claimed by another lane — the issue permits a hand-fed provider list for benchmarks, so id: holder discovery for blob hashes — provider table, gossip announcements, wire the blob gate #2260 is not a hard gate, but its holder-set API is the preferred feed and is in flight.queue/5810-2261-swarm-pocbehind the existing get path; (3) build the benchmark fixture (local multi-provider FsStore nodes) before the scheduler, so measurements exist from the first working fetch; (4) post the benchmark transcript on this issue and open the PoC PR as explicitly review-gated.132 remaining items
dezren39 commentedon Nov 3, 2026
🤖 Agent note · handed off
opencode(lane row already executed via a bound session on a prior cycle)dezren39 commentedon Nov 3, 2026
🤖 Agent note · blocked — upstream gate #2260 still open and escalated
aea12c82-c5af-4e5a-9b66-f8e360f87177)Why blocked, not parked
The packet resumes "after #2260 closes and its decision lands". Fresh check 2026-11-03T15:18Z: #2260 is OPEN with
status/blocked+queue/blocked+user/escalated(holder discovery / provider table / blob gate). No PR exists for #2261 (gh pr list --search 2261|swarm|chunk-range— none), so the swarm scheduler this PoC builds on has no landed holder-discovery design to target.This task has now been claimed and parked/cycled seven times on the same unmet gate (prior park 2026-11-03 12:44Z, handoff 13:45Z). Parking again just schedules an eighth cycle, so the task is blocked instead — it needs the #2260 decision, which is Drew's.
Resume condition (mechanical)
Unblock when #2260 closes with its decision landed. Then: revalidate #2261 at current
main, read #2050 (CLOSED design) sections 5–6 and the #2260 outcome, rungit merge-tree-style conflict checks, and implement the PoC in a worktree under~/git/fleet/— no new design work, this is the implementation of the already-settled design.No code, branch, or PR was produced by this claim.
developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 5810 (small_fix, blocked, priority 30) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03