investigate: GitHub secondary rate limits — fleet content-creation pacing so filing bursts stop blocking the account #2013

Activity

1. developing-today commented on Oct 29, 2026

   

   developing-todayBot

   on Oct 29, 2026 – with developing.today

   Contributor

   More actions

   CODE-934
2. dezren39 commented on Oct 29, 2026

   

   dezren39

   on Oct 29, 2026

   MemberAuthor

   More actions

   Parent epic: #2055 — fleet expansion & placement program (2026-10-28 directive). This issue is batch-2 item B11 of that program; it was filed first, as the probe that confirmed the content-creation limit had cleared.
3. github-actions

   assigned

   dezren39

   on Oct 29, 2026
4. dezren39

   added

   priority/highImportant and time-sensitiveImportant and time-sensitive

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   on Oct 29, 2026
5. dezren39 commented on Oct 29, 2026

   

   dezren39

   on Oct 29, 2026

   MemberAuthor

   More actions

   Top priority — ID program wave-1. Drew directed this be picked up in the next runs (2026-10-29): enqueued directly as triage task 3334 in repo-health-code-pilot at queue priority 20 (only breakage-class work at priority 10 outranks it). Wave-1 is the ungated investigations and designs; syntheses and PoCs stay gated on their named inputs. Investigation route: Workhorse.
6. dezren39

   added

   work/assignedQueue worker lifecycle stateQueue worker lifecycle state

   queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

   status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

   and removed

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   on Oct 29, 2026
7. dezren39 commented on Oct 29, 2026

   

   dezren39

   on Oct 29, 2026

   MemberAuthor

   More actions

   🤖 Agent note · picked up

   • Agent: queue worker (repo-worker-code), OpenChamber scheduled lane
   • Session / ID: queue task 3334 (triage)
   • Found via: queue claim, priority 20 (Drew-directed wave-1)
   • Task: Triage investigate: GitHub secondary rate limits — fleet content-creation pacing so filing bursts stop blocking the account #2013 — verify the secondary-rate-limit premise, measure fleet content-creation rates per lane/job, and produce the pacing design (per-job budgets, batching windows, shared throttle) plus the verify-by-read discipline for filing scripts.
   • Expectation: investigation/design recorded on this issue plus a routed follow-on this run; no permanent feature built here.
   • Links: epic: fleet expansion & placement program (2026-10-28 directive, LIST 1–5) #2055 (program epic) · epic: ID platform expansion — apps, comms, currency, hosting, boards (2026-10-28 directive, LIST 6–15) #2056 (ID epic) · task 3334
8. developing-today

   added

   status:in-review

   on Oct 29, 2026
9. dezren39 commented on Oct 29, 2026

   

   dezren39

   on Oct 29, 2026

   MemberAuthor

   More actions

   Investigation result

   Design record: PR #2226 — doc/2026-10-29T10-08-00Z_design_github-content-creation-pacing/. Follow-on implementation: #2227.

   Premise: confirmed, with measurements

   GitHub API, developing-today/code, bucketed by created_at on the laptop 2026-10-29 (issues + PRs from /issues?state=all, comments from /issues/comments):

   Day (UTC)	Issues + PRs	Comments	Total content-creating events
   2026-10-27	494	1 363	1 857
   2026-10-28	479	2 437	2 916
   2026-10-29¹	217	1 325	1 542

   ¹ through 10:00Z.

   • Peak hour 2026-10-28 22:00Z = 511 events (359 comments + 152 issues/PRs) — at GitHub's documented 500 content-generating requests per hour ceiling, for this one repository, before the other three fleet repos and any web-UI actions.
   • Peak minute: 39 (2026-10-29 01:00Z, the batch-2 filing). Ten-minute stretches reached 144, 129 and 95 — the 80/min ceiling was never hit, so the block came from the hourly, not the per-minute, limit.
   • Creation stopped after issue id(repl): find selection prompt silently drops unparseable numbers — a typo changes which blobs are fetched, and duplicates fetch twice #1999 at 23:08:35Z and resumed at 23:41:24Z — a 31m48s gap with work queued and the fleet active. The evening's prepared program filing did not complete; investigate: GitHub secondary rate limits — fleet content-creation pacing so filing bursts stop blocking the account #2013 landing at 00:58:54Z is consistent with being the probe after recovery.
   • One identity: 1 900 of 1 978 recently created items are dezren39. Every lane, producer and interactive agent shares the account, so the ceiling is a single account-wide budget nothing can see.
   • Comments are the volume: 2 437 on 28 October, five times the issue/PR count — agent notes at every transition.

   Caveat: the 403 itself and the gh issue create exit-0-with-no-output behaviour come from this issue's report; no client log of them exists on the laptop, and they were not reproduced. The rates, the gap and the composition are measured independently.

   Documented limits (verified 2026-10-29)

   ~80 content-generating requests/min and ~500/hour (web UI counts too; some endpoints lower); 100 concurrent requests; 900 points/min per REST endpoint; 90 s CPU per 60 s. No way to query secondary-limit status — x-ratelimit-* covers the primary limit only. A breach is 403/429 with optional retry-after; without it wait ≥ 60 s; continuing to retry "may result in the banning of your integration".

   Proposed design

   1. Shared creation ledger (~/.local/state/fleet/content-ledger.sqlite): one row per content-creating call, reservation before, outcome after; 60 s and 3 600 s windows read from the ledger, atomic across processes.
   2. Throttle wrapper: 45/min and 300/hour account-wide (~60 % of the ceiling), ≥ 1 s between POSTs, ≤ 8 in flight, per-lane 50/hour sustained with idle share reclaimable, 75/hour held for priority-10 breakage, bulk programs ≤ 20 issues per 30-minute window.
   3. Verify-by-read: success only when a read confirms the object — non-empty stdout with a number/URL, else a <!-- fleet-nonce:… --> search; empty output is a failure, never success. This is the fix for the silent no-op.
   4. Circuit breaker: honour retry-after, else 60 s then ×2 up to 5 attempts; then stop content creation for 15 minutes and park the calling task with a resume brief.
   5. Structural: one live agent note per task edited in place instead of one comment per checkpoint; target ≤ 2 comments per task.
   6. Gap detector: alert when filing work is queued and no creation happens for > 5 minutes.

   Rollout order: ledger + verify-by-read, then warn-only throttle for a week, then enforcing. Implementation is #2227; this issue can close once #2227 lands.
10. dezren39

    added a commit that references this issue on Oct 29, 2026

    Merge pull request #2226 from developing-today/queue/3334-2013-conten…

    ...

    1781911
11. developing-today

    added

    status:done

    and removed

    status:in-review

    on Oct 29, 2026
12. developing-today

    closed this as completedon Oct 29, 2026
13. dezren39 commented on Oct 29, 2026

    

    dezren39

    on Oct 29, 2026

    MemberAuthor

    More actions

    🤖 Agent note · done

    • Agent: queue worker (repo-worker-code), OpenChamber scheduled lane
    • Session / ID: queue task 3334 (triage)
    • Outcome: investigation + design delivered; follow-on routed.
      • Design record, merged: PR doc: fleet content-creation pacing — measured rates, the 2026-10-28 block, and a shared throttle (#2013) #2226 → doc/2026-10-29T10-08-00Z_design_github-content-creation-pacing/ (merge 1781911).
      • Implementation routed: impl: fleet content-creation throttle — shared ledger, pacing budgets, and verify-by-read filing #2227 — shared creation ledger, throttle with per-lane budgets and breakage reserve, verify-by-read filing, circuit breaker, one-live-note comment reduction, gap detector.
      • Measured findings in the comment above: 2 916 content-creating events on 2026-10-28 (peak hour 22Z = 511 = the documented 500/hour ceiling), 31m48s creation gap after id(repl): find selection prompt silently drops unparseable numbers — a typo changes which blobs are fetched, and duplicates fetch twice #1999, comments at 5× issue volume, one shared account identity.
    • Verified: design file present on main; treefmt on the changed file locally — 0 files changed. CI jobs (treefmt check, Auto Assign) were queued with no runner available at merge time; the repository has no required status checks and the change is one Markdown file.
    • Not verified: the 403 and the gh exit-0-with-no-output behaviour are from this issue's report; not reproduced (no client log found on the laptop).
    • Closing: this issue asked to investigate, measure and propose pacing. All three are delivered; further work is tracking on impl: fleet content-creation throttle — shared ledger, pacing budgets, and verify-by-read filing #2227.
    • Bookkeeping: sandbox backlog/sweep bookkeeping deferred to fleet reconciliation (OpenChamber scheduled lane).
14. dezren39

    removed

    status/in-progressQueue worker lifecycle stateQueue worker lifecycle state

    queue/claimedQueue worker lifecycle stateQueue worker lifecycle state

    work/assignedQueue worker lifecycle stateQueue worker lifecycle state

    on Oct 29, 2026