Repository navigation
fleet-feed producer (~/git/pm-feed) silently went stale: AA section vanished from the dashboard #1349
Description
What happened (2026-10-27, fixed ~02:29 CDT)
The AA explorer page (/aa + /api/aa.json, shipped in #1250) showed {"status":"absent","reason":"no AA explorer section in the fleet feed yet"} even though the aggregator deploy was current and a manual collector run at 00:29 CDT produced a perfect AA section (138 rankings / 1,144 thresholds / 35 configs / 183 test runs).
Root cause: the feed's actual producer is not the laptop deploy — it's the sandbox clone ~/git/pm-feed. fleet-feed-collector.timer (sandbox, ~5 min) runs ~/git/pm-feed/pm-stack/aggregator/scripts/push-fleet-feed.sh, which collects in that clone and scp's the result to the laptop. The clone sat at merge-of-#994 (14dd66b6); its collector (18,570 bytes) predates the AA section entirely — zero ainventory references — so every push overwrote the laptop feed with a section-less one. The 00:29 good run came from ~/pm-stack-update/aggregator (the deploy working copy), which the timer never executes.
Fix applied tonight (operational, not in this repo)
git merge --ff-only origin/mainin~/git/pm-feed(14dd66b6→3a32780c, includes feat(pm-aggregator): AA cost explorer page (fleet-feed backed) #1250). Collector is now the merged 25,633-byte version.- The clone is a sparse checkout (only
pm-stack/aggregator); the new collector reads.opencode/opencode.jsoncat the repo root for the AA coverage lists and failed soft ("opencode.jsonc unreadable; AA coverage omitted"). Added.opencodeto the sparse checkout. - Verified end-to-end: collector summary
aa 138 rankings / 35 configs / 183 test runs, feed pushed,/api/aa.json→status: okwith coverage (4 lists),/aa+ board HTTP 200.
Why this will recur
Nothing keeps ~/git/pm-feed current: laptop deploys (pm-stack-update + install.sh) refresh the server, but the producer clone only moves when a human fast-forwards it. It drifted ~250 PRs behind silently — the only symptom was a dashboard section going "absent", and the stale collector can't even log the new design's loud collector: AA section absent (...) error because it doesn't contain that code.
Proposed durable fix (pick one)
- Updater step: a tiny
pm-feed-synconeshot (or a pre-step ininstall-collector-units.sh's units) thatgit fetch+merge --ff-onlys the feed clone before collector runs — never auto-resolving, failing loud on divergence. Must not edit the running push script in place mid-run (pull in a separate unit, then start the collector). - Staleness guard in
push-fleet-feed.sh: before collecting, compare clone HEAD toorigin/main; if behind by more than N commits, log a loud journal error (and optionally stamp the feed with the producer commit so/apican surface "producer is stale"). - Feed provenance: embed the collector's git HEAD in the feed envelope (
generatedAtalready exists); the aggregator's absent/stale states could then name the producer commit.
Option 1 is probably the real fix; 2 is the cheapest loud tripwire. Also worth documenting in the aggregator README that the producer is the sandbox feed clone, not the laptop working copy — that fact cost most of the diagnosis time.
Acceptance criteria
- A durable mechanism (updater step, staleness guard, or equivalent) keeps the sandbox
~/git/pm-feedproducer clone current withorigin/main, or fails loud in the journal when it cannot — never silently pushing a section-less feed again. - The feed envelope or
/apisurface can name the producer commit, so a stale producer is diagnosable from the feed itself. - The aggregator README documents that the producer is the sandbox feed clone, not the laptop working copy.
- Verified end-to-end: after a simulated stale-clone state, the next collector cycle either self-heals or reports the staleness loudly, and
/api/aa.jsonreturnsstatus: ok.
Activity
developing-today commentedon Oct 27, 2026
CODE-451
dezren39 commentedon Oct 27, 2026
Intake review
🤖 Agent note · filed
Related: Related: the AA explorer shipped in #1250; the incident itself was fixed operationally tonight (clone fast-forwarded, sparse checkout widened) — this issue is the durable recurrence guard, and no merged PR resolves it.
Risks / notes: The producer being a sandbox clone rather than the laptop deploy copy is the trap that cost the diagnosis time; any fix that only updates the clone once will recur on the next aggregator change. Risk: an auto-update step must never auto-resolve divergence or edit the running push script mid-run.
Suggested approach: Option 1 from the body (a separate sync oneshot that fast-forwards the clone and fails loud on divergence), plus option 3's producer-commit stamp so the next stale state names itself.
Liveness: searched merged PRs and recent fleet records this run — no merged PR explicitly resolves this issue, so it stays open. Marked
meta/enriched.dezren39 commentedon Nov 1, 2026
agent-note:v1 — queue task 2559 (lane E) — triage verdict: CONFIRMED, recurrence condition live today
Verified 2026-11-01:
git -C ~/git/pm-feed rev-list --count HEAD..origin/main= 1470 commits behind (HEAD 93a8d8e, 2026-10-29), whilefleet-feed-collector.timeris active and firing (last cycle 04:39Z; /api/health shows the feed 3m old). The clone predates ~3 days of merged collector work and nothing notices./api/aa.jsonabsence is a different failure —Supabase query failed (HTTP 403)— already tracked by pm-aggregator /aa explorer data feed broken: Supabase query 403 after muse-dev2 migration #3294 and AA collector: a transient Supabase 403 wipes the explorer — keep last-good payload, skip query when batch unchanged #3332; it is not evidence for or against this issue, and this triage does not duplicate it.Route: scoped implementation packet enqueued as queue task 5006 (small_fix): pm-feed-sync ff-only step failing loud on divergence, producer HEAD stamped into the feed envelope and surfaced via /api, aggregator README documenting the sandbox-clone producer topology, verified with a simulated stale clone.
17 remaining items
dezren39 commentedon Nov 3, 2026
🤖 Agent note · picked up
dezren39 commentedon Nov 3, 2026
🤖 Agent note · blocked
packet_defect.generatedAtand no producer commit, but current main already emitscheckout.headfromcomputeCheckoutDrift()(FleetFeed.checkout?: CheckoutDrift). The real API gap is that/api/healthomitscheckout; README producer-host rows still claim sandbox. The packet must be corrected to reusecheckout.headand state whether HEAD must remain present whenorigin/mainis unreadable, rather than creating a duplicate field.queue/5006-1349-fleet-feed-commit-stampretained.developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 5006 (small_fix, blocked, priority 30) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 5006 (small_fix, blocked, priority 30) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03