Repository navigation
impl: fleet content-creation throttle — shared ledger, pacing budgets, and verify-by-read filing #2227
Description
Implementation for the design in #2013 — design record: doc/2026-10-29T10-08-00Z_design_github-content-creation-pacing/ (PR #2226).
Problem
Every fleet lane authenticates as one account, so GitHub's content-creation ceiling (documented: ~80/min, 500/hour) is a single shared budget no job can see. On 2026-10-28 developing-today/code alone produced 2 916 content-creating events (479 issues/PRs + 2 437 comments), peaking at 511 in the 22:00Z hour; creation then stopped for 31m48s after #1999. gh issue create exited 0 with empty output while creating nothing, so no caller noticed.
Scope
Code lands with the fleet tooling (~/hatch/tools, repo-queue.py, producer/filing scripts), not in this repository. This issue is the tracking record and the spec pointer.
- Creation ledger —
~/.local/state/fleet/content-ledger.sqlite, one row per content-creating call (ts_utc, actor, job, repo, kind, ref, outcome), written as a reservation before the call and finalised after the verifying read; windows computed from the ledger (60 s, 3 600 s), atomic across processes viaBEGIN IMMEDIATE. - Throttle wrapper —
reserve(job, kind)before every create. Account-wide 45/min and 300/hour; ≥ 1 s between POSTs; ≤ 8 in flight; per-lane 50/hour sustained with idle share reclaimable; 75/hour reserve for priority-10 breakage work; bulk programs ≤ 20 issues per 30-minute window. - Verify-by-read — success only when a read confirms the object: require non-empty stdout with a number/URL, else search for the
<!-- fleet-nonce:… -->marker in the body; recordcreated | verified_existing | failed. Empty output is a failure, never a success. - Circuit breaker — honour
retry-after; without it wait 60 s then exponential ×2 (5 attempts); after 5 failures stop content creation for 15 minutes, park the calling task with a resume brief, alert. Never spin (GitHub warns retrying while limited can ban the integration). - Structural reduction — one live agent note per task, edited in place, instead of one comment per checkpoint (comments are 5× the issue volume). Target ≤ 2 comments per task.
- Gap detector — alert when queued filing work exists and no creation is seen for > 5 minutes (belongs in the fleet monitor).
Acceptance
- A filing script run through the wrapper cannot exceed the throttle budgets, proven by a test that replays a 100-create batch and asserts the ledger shape.
- A simulated empty-output/403 create is recorded as
failed, and the batch tally does not count it — regression test for the 2026-10-28 silent failure. - A simulated 403 with
retry-afterand one without both back off per the rules, and five failures open the circuit. - Warn-only run for one week: the throttle logs what it would have delayed without delaying, and the log is reviewed before enforcing.
- Success measure after enforcement: no creation 403 for 14 days, account-wide hourly total under 300, and queue throughput unchanged.
Order
Ledger + verify-by-read first (additive, fixes the silent-success bug), then the throttle in warn-only mode, then enforcing, then the comment-reduction change in the note-writing path.
Out of scope
- The 42-issue program's partial filing (unknown whether items were lost; the ledger makes it answerable next time).
- Multi-host coordination (open question in the design if a sandbox lane ever files).
Acceptance criteria
- The creation ledger exists at the design's path and records every fleet content-creating call (issues, PRs, comments) with actor/job/repo/kind/ref/outcome, written as a reservation before the call.
- Pacing budgets from the design (investigate: GitHub secondary rate limits — fleet content-creation pacing so filing bursts stop blocking the account #2013) are enforced at the shared choke point(s): a caller that would exceed the budget is delayed or visibly refused — never silently dropped — and the
gh issue createexits-0-but-creates-nothing failure mode is detected and surfaced to the caller. - A verify-by-read pass demonstrates the ledger's counts against GitHub's actual creation events for one busy hour (the 2026-10-28 22:00Z peak hour, 511 events, is the natural test case).
Metadata
Metadata
Assignees
Labels
Large: multi-day or open-ended; expect staged check-insNew feature or requestInfrastructure / host-portability workIntake review done: acceptance criteria, links, and normalized labels addedImportant and time-sensitiveFiled by the new-issue-creator jobQueue packet kind: verify, reproduce, and route this item
Activity
developing-today commentedon Oct 29, 2026
CODE-1100
dezren39 commentedon Oct 29, 2026
Intake review
effort/landmeta/enrichedadded.developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7310 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: picked-up
task: 7310
worker: repo-worker-code
lane: 335a5d1f-bfc1-4831-99ea-850c8939a727
origin: openchamber-scheduled-lane
next: Revalidate current state of issue #2227 and inspect the implementation on current main before deciding whether it is done, blocked, needs a split, or needs implementation.
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: done (triaged)
task: 7310
worker: repo-worker-code
lane: 335a5d1f-bfc1-4831-99ea-850c8939a727
origin: openchamber-scheduled-lane
findings: #2227 remains open; design/investigation #2013 is closed completed and design PR #2226 is merged. Fresh GitHub checks found no linked implementation PR or matching PR. The current ~/hatch source checkout has no ledger, pacing, or verify-by-read implementation markers outside historical state notes. The historical 2026-10-28 403/empty-output behavior was not reproduced; the earlier investigation records no client log.
blocker: the #2227 spec points implementation at ~/hatch, whose GitHub source is dezren39/hatch, not developing-today/code. This repo-health-code-pilot lane has no hatch execution queue, so this lane did not edit hatch source.
fix-path: opened phase-one execution issue dezren39/hatch#3 for the shared ledger and verify-by-read, with later throttle/circuit-breaker/comment-reduction phases left under #2227.
next: execute dezren39/hatch#3; keep #2227 as the tracking/spec pointer for remaining ordered phases.