Skip to content

pm-stack(aggregator): /board, /api/board.json, /api/overview.json + /frag/overview die with empty replies at ~12s; /program HTML regrown to 1.16MB #5082

Open

Description

@developing-today

Filed by the pool-quota-steward follow-through (Drew, 2026-11-03: "/program and board are both fucked up and something needs to watch them; it might as well be steward"). The steward now checks these surfaces every run and will refresh this issue with fresh observations; this filing carries the first evidence set.

Symptoms (laptop, aggregator on 127.0.0.1:4870, direct Bun server — no proxy in front)

  • GET /board and GET /api/board.json: intermittent empty replies — the connection closes with no bytes after ~12.0s. Observed failing repeatedly on 2026-11-03 (morning through 13:1x CST); at 12:53 CST both returned 200 (59,918B / 9,841B) but took 5.1s; at ~13:12 CST both died again at 12.001s / 12.002s (curl exit 52, 0 bytes).
  • GET /api/overview.json and GET /frag/overview: same failure family — empty reply at 12.000s (observed 13:09–13:12 CST). The Overview page shell (/) serves instantly, but its quota strip arrives via /frag/overview, so the front door shows the same breakage.
  • GET /program: serves 200 fast (0.82s) but the HTML payload is 1,165,151 bytes — the slimming landed earlier (44.3MB → 716KB) has regressed by ~63%. /api/program.json is 65,931B and fast.

Failure point (as far as identified)

  • The dying routes share one assembly: /api/overview.json is deriveOverview(await overviewContext()) (src/server.ts:1522) and /frag/overview uses the same overviewContext(); /board + /api/board.json render the board facet of the same context family. Quota-only routes (/api/quota.json, /quota) answer in ~0.03s, so the quota path is NOT the slow part.
  • The cutoff is suspiciously exact (~12.000s, connection closed, zero bytes): the handlers produce no bytes while assembling, and something in the serving path gives up at ~12s. Bun.serve is configured with no explicit idleTimeout (src/server.ts:1928). Whether the 12s is Bun's idle window or a client-visible sum of awaited sub-timeouts (the projector facet at 127.0.0.1:7801 has an 8s timeout; projects.ts uses 20s; supabase.ts 60s), the assembly intermittently exceeds it — at 5.1s on a good run, >12s on a bad one — and the failure mode is silent (empty reply, nothing in the response to diagnose with).
  • Latency variance suggests cache/load dependence in the facet assembly; the steward's hourly checks will show the failure cadence.

What is NOT being asked here

No rebuild in this issue: the steward WATCHES and reports (state + latency + bytes for /program, /api/program.json, /board, /api/board.json every run, recorded in its report; broken state named in its delivery). This issue is the evidence home. Related: #1503 (shell pages must surface stalled fragments — adjacent, about fragment UX rather than the assembly latency itself).

Acceptance shape (for whoever picks it up)

/board, /api/board.json, /api/overview.json and /frag/overview answer 200 with content across repeated probes, including cold ones; /program HTML payload back near its post-slimming size; a slow assembly degrades to a visible stale/error state instead of an empty reply.

Activity

  1. dezren39 commented on Nov 4, 2026

    @dezren39
    Member

    Pool-quota steward, 139th run (2026-11-04) — the intermittent empty-reply failure REPRODUCED this run, twice, on independent probes (all laptop aggregator, localhost:4870):

    1. Full-surface batch probe (~03:28Z): /program 000 after 12.002s (0 B), /board 000 after 12.008s (0 B), /api/board.json 000 after 12.005s (0 B), /frag/overview 000 after 12.009s (0 B), /api/overview.json 000 after 12.008s (0 B) — all five board/overview-assembly surfaces dead simultaneously at the ~12s mark, while /api/quota.json (200, 0.20s), / (200), /quota (200), /today (200), /aa (200) and /quota/service/turso (200) answered normally in the same batch. /api/program.json answered but slowly (200, 8.12s, 64,896 B).
    2. Independent re-probe (03:34Z): /api/board.json 000 after 12.011s (0 B) while /program (200, 1.63s, 1,167,098 B), /api/program.json (200, 0.11s), /board (200, 5.17s, 60,287 B), /frag/overview (200, 9.74s — slow) and /api/overview.json (200, 5.06s) answered.
    3. Immediate follow-up (03:34Z): /api/board.json answered 200 twice at ~5.1s (9,841 B) — confirming intermittency, not a hard outage.

    Pattern matches this issue exactly: the shared overview/board assembly intermittently dies with empty replies at ~12s. Note: an earlier probe result in the same steward run reported all surfaces healthy — the failure is flapping within minutes, so single healthy probes do not clear it. Steward is watch-and-surface only per its instruction; no aggregator repair attempted. Separately, /program HTML remains ~1.167 MB.

  2. dezren39 commented on Nov 4, 2026

    @dezren39
    Member

    Steward refresh — 2026-11-04 daily pass (141st run, ~06:19Z, laptop localhost:4870): the empty-reply failure mode REPRODUCED on all four board/overview-assembly surfaces in one probe batch — /board 000 (no answer within the 25s probe cap, 0 B), /api/board.json 000, /frag/overview 000, /api/overview.json 000. In the same batch: /program 200 @ 3.16s (1,167,043 B — the ~1.167 MB HTML size concern stands), /api/program.json 200 @ 0.11s (64,896 B), and every quota surface answered 200 (/ 4.98s, /quota 5.02s, /today 3.77s, /aa 2.24s, /quota/service/turso 5.13s, /api/quota.json 5.08s 51,808 B). This matches the intermittent pattern recorded 2026-11-04T03:28–03:34Z: the shared board/overview assembly dies while program and quota surfaces stay up. Steward is watch-and-surface only; no repair attempted.

  3. dezren39 commented on Nov 4, 2026

    @dezren39
    Member

    Steward addendum to the 141st-run refresh above (comment 6624171213): a second probe batch from the same run (~06:2xZ, laptop localhost:4870) returned a DIFFERENT failure mix, and a steward re-probe at ~07:05Z found everything healthy — the surfaces are flapping within minutes, exactly this issue's intermittent mode. Batch 2: /program 000 @ 12.00s (0 B — the empty-reply death reached /program itself this time), /board 000 @ 12.00s, /api/board.json 000 @ 12.00s, /api/program.json 200 but slow @ 8.06s (61,773 B), /frag/overview 200 slow @ 9.76s, /api/overview.json 200 @ 5.03s; quota surfaces all fast 200 in that batch. Re-probe ~07:05Z: /program 200 @ 1.90s (1,166,369 B), /board 200 @ 5.18s, /api/board.json 200 @ 5.03s, /frag/overview 200 @ 5.33s, /api/overview.json 200 @ 5.04s. Combined picture for the run: batch 1 = board assembly four surfaces dead, program up; batch 2 = program + board + board.json dead, overview fragments up slow; re-probe = all up, board assembly ~5s. Watch-and-surface only; no repair attempted.

  4. dezren39 commented on Nov 4, 2026

    @dezren39
    Member

    🤖 Agent note · handed off

  5. added
    queue/queuedHas a queued task in the Supabase agent queue
    task/triageQueue packet kind: verify, reproduce, and route this item
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 4, 2026
  6. dezren39 commented on Nov 4, 2026

    @dezren39
    Member

    agent-note:v1
    {"kind":"handoff","task":9456,"lane":"code-v","execution_lane":"opencode","state":"parked","note":"task 9456 (triage) handed off to opencode claimant by lane code-v. Fresh revalidation: issue #5082 OPEN, no hard-stop labels, no prior task progress — claimant triages the pm-stack aggregator empty-reply deaths (/board, /api/board.json, /api/overview.json, /frag/overview at ~12s) from scratch."}

  7. added
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    and removed on Nov 4, 2026
  8. 4 remaining items

  9. added
    queue/claimedQueue worker lifecycle state
    and removed
    status/parkedDeliberately paused with a recorded resume condition
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    on Nov 4, 2026
  10. developing-today commented on Nov 4, 2026

    @developing-today
    ContributorAuthor

    🤖 Agent note · handed off

    • Agent: OpenCode claimant (opencode-claimant)
    • Queue task: task 9456 (repo-health-code-pilot, worker repo-worker-code)
    • Issue: #5082
    • Session: ses_e5e58645820ffegUkORUEaZiFM
    • Route: opencode-route/workhorse; agent build
    • Expectation: this bound session works only task 9456, revalidates fresh state, heartbeats, and records a terminal queue state with this same session id.
  11. dezren39 commented on Nov 4, 2026

    @dezren39
    Member

    Lane M claimed task 9456 (triage) for #5082; execution-lane assignment returned opencode, so this task is handed off to the laptop claimant (opencode-claimant), which launches the bound session. Attempts 9/12.

  12. dezren39 commented on Nov 4, 2026

    @dezren39
    Member

    🤖 Agent note · picked up

  13. dezren39 commented on Nov 4, 2026

    @dezren39
    Member

    🤖 Agent note · triaged

  14. dezren39 commented on Nov 4, 2026

    @dezren39
    Member

    🤖 Agent note · done

  15. dezren39 commented on Nov 5, 2026

    @dezren39
    Member

    Pool-quota steward run 160 (2026-11-05T00:31Z / 18:31 CST) — step-6 surface watch, first run after the laptop tunnel recovered:

    • /board — BROKEN: empty reply at 12.003s (curl HTTP 000, 0B)
    • /api/board.json — BROKEN: empty reply at 12.002s (0B)
    • /frag/overview — BROKEN: empty reply at 12.002s (0B) — so the Overview quota strip is dead even though / itself serves (200, 0.001s, 45,893B static shell)
    • /api/overview.json — 200 this run (1.22s, 17,793B), recovered vs earlier observations
    • /program — 200, 0.59s, 1,169,554B (HTML still ~1.16MB)
    • /api/program.json — 200, 0.02s, 64,684B
    • /today 200 (1.56s, 46,258B); /quota 200 (0.02s, 69,998B); /aa 200 (539,654B)

    The board/overview-fragment assembly still dies at ~12s exactly as tracked here. Steward is watch-and-surface only on this issue.

  16. dezren39 commented on Nov 5, 2026

    @dezren39
    Member

    Steward watch refresh (pool-quota steward, 167th run, 2026-11-05 ~07:25–07:35Z):

    • /frag/overview — BROKEN on first fetch: HTTP 000, empty reply after 12.00s (the standing ~12s death mode on the overview/board assembly). Immediate retry: 200 in 11.29s (15,774B) — so the surface is intermittently dying again, not hard-down; even the successful retry sat just under the ~12s cliff.
    • Latencies on the sibling surfaces were sharply elevated in the same pass (vs 1.4–1.9s at the 06:27Z steward check): /board 200 in 8.53s (350,579B), retry 7.70s; /api/board.json 200 in 5.60s (1,780,674B); /api/overview.json 200 in 8.26s (15,385B); /program 200 in 5.38s (1,170,132B — HTML still ~1.17MB); /api/program.json 200 in 0.23s (63,319B).
    • Context, not a claimed root cause: laptop load average was ~34 at the time of the checks.

    Steward is watch-and-surface only for this path; no repair attempted.

  17. dezren39 commented on Nov 5, 2026

    @dezren39
    Member

    Steward observation — 168th pool-quota-steward run, 2026-11-05T08:24Z (02:24 CST):

    • /frag/overview BROKEN on first fetch again: HTTP 000, empty reply at 12.26s (curl rc=52 — the standing death mode). Retry succeeded: 200 at 7.72s, 15,402B. Intermittent, not recovered — the first-fetch death recurred after the 167th run's identical observation.
    • Sibling latencies remain elevated vs the 166th run's 1.4–1.9s baseline (laptop load ~50 at check time, context only): /board 200 (9.38s, 351,710B), /api/board.json 200 (7.46s, 1,793,378B), /api/overview.json 200 (7.96s, 14,989B), /program 200 (6.59s, 1,170,443B — HTML still ~1.17MB).
    • Healthy/fast: /api/program.json 200 (0.27s, 63,667B), /api/quota.json 200 (0.48s, 53,836B), /quota 200, /today 200, / 200, /aa 200.

    Watch-and-surface only (steward does not repair the board/aggregator path).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

task/triageQueue packet kind: verify, reproduce, and route this item

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions