Repository navigation
fleet-feed: unidentified stale-code collector overwrites the laptop feed every ~5-10 min (AA blanked, sessions undercounted) #3362
Description
Symptom
Since the collector moved to the laptop (2026-10-31), the laptop's fleet-feed.json (~/.local/state/pm-aggregator/fleet-feed.json) is periodically overwritten by a feed whose AA section is absent (reason: Supabase query failed HTTP 403) and whose session counts are ~24 instead of the laptop collector's 88. The explorer page blanks until the next laptop collector tick restores a good feed.
Verified timeline (2026-11-01 CDT)
- Good service writes: 00:30:09 (full build, aa-last-good store written), 00:35:09, 00:39:58, 00:54:38 — AA ok, 84-88 sessions.
- Phantom writes (top-level generatedAt): 00:31:50, 00:37:08, 00:47:12 — AA absent/403, attention 24.
- The phantom write is NOT the laptop systemd service (no journal entries at those times; the timer's LAST/NEXT do not match) and NOT the current code: PR aa feed: keep last good on failure + skip unchanged query; v_latest_import_batch rename #3349's keep-last-good collector degrades to the stored section on failure (forced-failure test 2026-11-01 passed: bogus project ref -> section ok + stale:true). The phantom produces absent, i.e. it runs pre-aa feed: keep last good on failure + skip unchanged query; v_latest_import_batch rename #3349 code or an equivalent old copy.
- A 7-minute process watch on the laptop (00:49:50-00:56:51) caught only the legitimate service tick; no phantom write landed in the window. SSH logins from the sandbox tailnet peer (100.115.16.3) at 00:32:03 and 00:37:22 correlate with push-sized gaps after the 00:31:50 / 00:37:08 collects. No sandbox cron, systemd timer, or loop process has been found yet. No phantom write observed since 00:47:12.
- Its file-based sections (projects 351, charges 75, DNS) are complete, so it reads the synced hatch-personal tree; its session source sees only ~24 sessions.
Ask
Identify the launcher (candidates: a leftover sandbox-era push loop, a lane/test session running a stale checkout's collector against the live feed path, an unretired unit somewhere) and stop it. A 30-min watcher is logging feed changes + collector processes on the laptop from 00:57 CDT (/tmp/phantom-watch2.log). Related: the collector's own units are hand-written and not yet in the pm-stack NixOS module (separate issue).
Acceptance criteria
- The phantom writer is identified by name (process/unit/copy of the collector code) with evidence — not just ruled out as the laptop systemd service or current main.
- The phantom writer is stopped/removed or updated to current code, and over a >=1h observation window every fleet-feed.json write carries the AA section (or the keep-last-good stale marker) and full session counts — no absent-AA overwrite recurs.
- The finding and the fix are recorded on this issue with timestamps from the feed file, so the next stale-copy incident has a named suspect list.
Activity
dezren39 commentedon Nov 1, 2026
Intake review
Related: #3363 (fold the collector into the NixOS module — removes the hand-installed copy class), PR #3349 (keep-last-good collector; the phantom predates it), #3332 (the AA 403 wipe this resembles — same blanking symptom, different writer). Notes: the body's timeline already rules out the laptop service and current code; the suspect is a stale copy running somewhere with write access to the laptop state dir. Risk: until identified, every ~5-10 min the explorer blanks — user-visible. Suggested approach: watch the file's writer via audit/fanotify on the laptop during a phantom window, and inventory collector copies on both machines.
— Fleet monitor intake pass (protocol §5), 2026-11-01. Still live at review time: no merged PR or commit found that resolves this item (checked against the issue's own evidence and recency).
fix(aggregator): retire the sandbox fleet-feed collector units (#3362)
Merge pull request #3865 from developing-today/fix/3362-retire-sandbo…
dezren39 commentedon Nov 1, 2026
Phantom identified — and stopped (2026-11-01)
The phantom writer is the sandbox host's own
fleet-feed-collector.timer(systemd, system level — which is why the earlier user-unit/cron hunts missed it), runningscripts/push-fleet-feed.shfrom the stale~/git/pm-feedclone (HEAD93a8d8ea, 2026-10-29 — pre-keep-last-good #3349, pre muse-dev2 pin).~/tools/init.shreinstalled its units on every sandbox boot. SSH logins from 100.115.16.3 at push times were this script's scp push (dest~/pm-stack/aggregatorsymlinks into the aggregator state dir).Why its AA section 403s (reproduced directly from the sandbox, token by name only —
supabase.access-token-muse-cli):nltgxorqwxhgfzseiirt(the stale code's default; its installed unit predates 5885986) -> HTTP 403{"message":"Your account does not have the necessary privileges to access this endpoint."}keihfchslgjetkmnmzra-> HTTP 201, current ainventory batchfe02783b-046c-4401-bb35-8c529b216b8dSandbox journal, every ~5 min:
collector: AA section absent (Supabase query failed (HTTP 403))->push-fleet-feed: feed pushed to laptop. Its other sections were also degraded (sandbox OpenCode server unreachable -> jules-only sessions ~14, unfiltered pools 11), racing the laptop collector (aa ok, 278 rankings, 74 sessions) — last writer won, so /aa blanked most of the time. This is also the live half of #3294.Fix
d8ead565cc46be22a0c6a79d9c912a62f4579691): deletes the sandbox-sidesystemd/fleet-feed-collector.{service,timer}+scripts/install-collector-units.shfrom the repo, markspush-fleet-feed.shmanual-only (kept for its charges confirm-after-delivery contract), and corrects the stale "sandbox collector" descriptions in README/feed.ts/topology.ts/watchdog.ts. Aggregator tests: 870 pass / 2 fail, identical on pristine main (pre-existing)./etc/systemd/system/fleet-feed-collector.{service,timer}removed, and the~/tools/init.shrestore block replaced with a retirement note so a sandbox recycle cannot resurrect it.Verification (2026-11-01, UTC)
ok, 278 rankings, 74 attention, pools filtered.GET /api/aa.json->{"status":"ok", batch fe02783b...}; /aa footer provenance:live (fleet feed · ainventory batch fe02783b · collected 2026-11-01T14:43:20.529Z)— no 403.Remaining acceptance item: the >=1h all-writes-good observation window — ~15 min verified here (17:10–17:25Z, two laptop ticks, two dead sandbox windows); the window completes ~18:10Z if someone wants to close this on the full hour.
(This comment replaces an earlier version whose inline-code spans were stripped by shell quoting during posting.)
developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7387 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
dezren39 commentedon Nov 3, 2026
agent-note:v1
event: picked_up
task: 7387 (repo-health-code-pilot, repo-worker-code; OpenChamber scheduled lane; session ses_e5e5c3748bbffeCrbxVyjhADa8)
state: Atomic claim recorded with origin=openchamber-scheduled-lane. Fresh #3362 is OPEN with no hard-stop labels; merged PR #3865 is linked. Existing issue comment reports the sandbox system-level fleet-feed-collector.timer was identified and stopped, so this triage will verify the remaining observation/acceptance evidence rather than repeat the original hunt.
next: Read the full resolution evidence and verify current status before deciding whether #3362 is already complete or needs a follow-up.
links: #3362 · #3865 · task 7387
dezren39 commentedon Nov 4, 2026
Current triage — 2026-11-03
The original phantom writer is identified: the sandbox system-level
fleet-feed-collector.timerranscripts/push-fleet-feed.shfrom stale~/git/pm-feedand pushed AA-absent/403 data to the laptop. Its retirement is implemented by PR #3865, now MERGED (CI SUCCESS); the prior activation report says the sandbox timer/service were stopped and removed and the~/tools/init.shrestoration block was retired.Fresh laptop evidence: the user
fleet-feed-collector.timeris active.journalctl --user -u fleet-feed-collector.service --since '2026-11-03 16:55:00'covers 16:55:30–17:55:59 CDT (>1 hour, 13 scheduled writes). Every logged write reports AA 283 rankings / 35 configs / 183 test runs and 64 sessions; the current feed at 17:55:34Z hasaa.status=ok, 283 rankings, and 64 attention entries.The journal proves the scheduled collector's writes were healthy, but does not observe arbitrary external changes between ticks. The only file watcher evidence (
~/tmp/phantom-watch2.log) is the earlier 17-minute window. Therefore I am not closing #3362 on the current evidence: the remaining blocker is direct observation of every file change for >=1 hour. A follow-up task will capture sanitized per-write timestamps, AA status, rankings, and session counts; close #3362 only if the window is clean, otherwise identify and route any new writer.dezren39 commentedon Nov 4, 2026
agent-note:v1
event: picked_up
task: 7387 (repo-health-code-pilot, repo-worker-code; OpenChamber scheduled lane 8ac61616-3815-4e7b-b33d-840a3829154a; session ses_e5e5b4941a0ffe0sZhl0lzt4kE)
state: Fresh #3362 is OPEN, labels already reflect claimed/in-progress, and linked PR #3865 is merged. Existing triage records the sandbox timer fix and a healthy laptop journal window, but says direct observation of every feed-file change for >=1 hour remains the acceptance blocker.
next: Verify the latest issue/PR resolution evidence and determine whether a tracked follow-up already covers the remaining observation; avoid repeating the original writer hunt.
links: #3362 · #3865 · task 7387
dezren39 commentedon Nov 4, 2026
agent-note:v1
event: triaged
task: 7387 (repo-health-code-pilot, repo-worker-code; OpenChamber scheduled lane 8ac61616-3815-4e7b-b33d-840a3829154a; session ses_e5e5b4941a0ffe0sZhl0lzt4kE)
state: Root cause confirmed in the existing resolution evidence: sandbox system-level
fleet-feed-collector.timerran stale~/git/pm-feedcode and pushed the AA-absent feed. PR #3865 merged at d8ead56 and retired the sandbox producer/boot restoration. Existing laptop journal evidence is healthy for >1h of scheduled writes. The localphantom-watch2.logends 2026-11-01 01:27:28 -05 after ~17 minutes; it does not establish the acceptance criterion of direct observation of every file change over >=1h. Therefore this issue remains OPEN for that final evidence; no new competing issue was found in the search.next: Capture a >=1h file-change observation with timestamps, AA status/rankings and session counts; close only if every write is healthy.
links: #3362 · #3865 · task 7387