Repository navigation
design: telemetry rollup — one stats pipeline under fleet init (sysstat + extra stats); init reduced to stats + prewarm #5925
Description
design: telemetry rollup — one stats pipeline under fleet init (sysstat base + fleet extra stats); init reduced to stats + prewarm
Framing — Drew's two-purpose test (2026-11-04 20:48 CST)
"now that init runs as part of every job starting, having an init job at all maybe is only helpful for getting stats and prewarming fleet conns."
So the test this design must pass: an init job earns its existence only by (a) stats collection and (b) prewarming fleet connections. Anything in today's init beyond those two is a candidate for deletion or re-homing, and the end state must say explicitly what remains of the boot-init job. This follows WS-C's init fix (75 s → 9 s), whose core lesson was the anti-pattern this design retires permanently: a live probe nobody consumes (the 60 s du that read "unknown" 138 times).
Inventory — what exists now (measured/read 2026-11-04)
| # | Component | Measures | Consumer | Cadence | Cost | Verdict |
|---|---|---|---|---|---|---|
| 1 | sysstat-collector (sadc -S ALL, looped 5×120 s runs → ~/logs/sysstat/sa-<day>, 14-day prune; restored by init.sh) |
CPU split (user/system/iowait/steal), memory, swap, per-device disk, network-interface aggregates | Incident forensics (#5701), WS-C attribution | 2 min samples | ~0.003% of a core (measured) | KEEP — the base collector |
| 2 | resource-sampler (~/logs/resource/resource-<day>.log) |
Load 1/5/15, mem used/avail, top process by cumulative %CPU, hatch daemon PID watch (death post-mortems) | Incident forensics | 120 s lines | Negligible CPU; a bespoke script + unit forever | FOLD — its unique values are the daemon-PID event and a crude top-process proxy; both are done properly by the rollup (pidstat history + a daemon-state event). Retire the script once those exist; keep historical logs |
| 3 | pidstat (sysstat suite) | Per-process CPU/mem/IO — the "which PID ate the box" history | Nobody — installed and working (verified live) but never run as a collector | — | — | ADOPT as the per-process layer of the rollup (see coverage analysis) |
| 4 | init.sh telemetry block | cache.nixos.org latency (HEAD + 3 parallel samples), nix version, store-path count, profile-bin count, subscription/quota status line → ~/logs/init-status (latest boot) + ~/logs/init.jsonl (per-boot records) |
Boot-init sweeps, metrics enumeration | Per boot | ~2–4 s of the 9 s init | FOLD into the rollup as the boot record (below) |
| 5 | Detached du → ~/logs/.nix-persist-size cache file |
nix-persist tree size, written detached post-WS-C; consumers read the cache | init.jsonl record | Per boot, detached | One background du per boot |
KEEP the pattern, generalize it — tree-size is not a sysstat metric; one throttled cache-writer per watched tree, always consumed with its age attached |
| 6 | trips.jsonl (client + daemon sides) | Per fleet-proxy trip: path, total_ms, status / duration, exit, byte counts | #5701/#5702 latency work, WS-C derived stats | Per call (1,644 daemon lines at last read) | Append-only local writes | KEEP raw; FOLD aggregates into the rollup (per-hour counts by status/failure class; per-leg p50/p95 once #5702's four timestamps land) |
| 7 | quota-recorder (1-min timer, ~5 s/run, broker-shipped records + Supabase acks) | Provider quota snapshots → stewardship visibility JSONL | Stewardship, quota-check | 1 min | The box's most frequent timer (#5775 has the cadence ruling) | KEEP SEPARATE — it is a billing-evidence record about external providers with delivery/ack semantics, not host telemetry; coupling it to a host-stats pipeline buys nothing and risks the record. Rollup consumers may read its outputs as a source |
| 8 | fleet-watchdog per-lane reads | Lane consecutive-failure counts, last-good times | Watchdog alerts | ~10–15 min | Repeated live joins across cron runs × agent.tasks | CONSUMER, not a collector — the per-lane proof gauge (tasks completed per lane per shift; the join fields exist on both sides, nobody computes it) becomes a rollup output the watchdog reads |
| 9 | conn-watch.py | (was) connection watching | Nobody | — | A zombie holding a deleted-inode fd until WS-C killed it | DELETE remnants — already stopped (#5775 records it); remove the script so nothing resurrects it |
| 10 | init.sh live verification probes (services-verify 900 s budget, go/rust presence checks, nix eval smoke) |
"Is it healthy?" — answered synchronously at boot | Nobody, since WS-C | Per boot | The synchronous-probe anti-pattern in its purest form | DELETE from init — health is read from the rollup/watchdog, never probed at boot |
| 11 | init.sh restorers (per-service install.sh unit installs incl. #1/#2 above; nix-ensure/wrapper/profile-sync; symlink-layer repair) |
Not stats. Not prewarm. | The sandbox recycle cycle | Per boot | Most of init's remaining bulk | MOVE OUT — they don't die, they change owner: the generic restorer (#5770, deploy design) + platform boot. Init is the wrong home (see two-purpose audit) |
Coverage analysis — what sysstat already covers, what it cannot
Measured against the actual collector invocation (sadc -S ALL 120 5, looped):
- Covered fully: system aggregates — CPU split including iowait and steal (the Incident record 2026-11-04: afternoon responsiveness loss — sandbox wedged after post-boot herd (daemon restart, 88-min sampler silence); laptop path exonerated 21:00–21:28 UTC #5701 oversubscription signal), memory/swap, per-block-device throughput and utilization, per-interface network throughput/errors.
sar/sadfread the daily files; 14 days of history already accumulate (~2 MB/day). - NOT covered by sadc, by design: per-process anything. sadc has no per-PID activity; the sa files can never answer "which process." pidstat is in the same suite, installed, and verified working on this box — it is simply never run persistently. The rollup therefore adds a pidstat collector loop (same unit family, e.g.
pidstat -urd -p ALLat the sadc cadence, daily files beside the sa files) and the resource-sampler retires. Open choice for the implementer: 120 s to match sadc, or 60 s for finer incident resolution — storage is trivial either way. - NOT a sysstat metric at all: directory-tree usage (
du). Handled by the generalized cache-writer pattern (Bump build/buck/preludes/buck2-prelude from3a02ea3to370cd4d#5): detached, throttled, value+timestamp, consumers tolerate staleness explicitly. - Filesystem fill: present in the boot record only (
disk_home). Ifsar -Fon our sa files doesn't yield a fill series, add a one-linedfsample to the rollup writer rather than a new probe job.
The two-purpose audit of today's init
init-ensure.sh (runs in every job's prelude — the "init runs as part of every job starting" of Drew's framing):
- Session-proxy ensure (
fleet proxy serve,LAPTOP_AGENT_POOL=50) → PREWARM. Keep — this becomesfleet init's sandbox role per design: fleet init location awareness — role manifest, env overrides, report-don't-guess failure rule #5687 (spawn {proxy}, pool warm), so the first real trip never pays cold CONNECT costs (cold 1.2–1.8 s vs warm p50 369 ms, WS-C). - Fleet symlink ensure (
~/tools/bin/fleet,repo-queue) → neither purpose; provisioning residue. Keep as a cheap guard only until the deploy tool (deploy tool v1: target registry (deployTargets) + systemd generation activation #5766) owns links, then delete.
init.sh (the boot job): telemetry block → STATS (folds, item #4). Everything else — restorers (#11), verification probes (#10) → neither purpose: re-homed or deleted per the inventory.
End state
fleet init (location-aware, #5687) ensures exactly: {proxy serve + pool prewarm; sysstat-collector; pidstat collector; tree-size cache writers} — on the sandbox. On the laptop it ensures its own serve roles as #5687 specifies; the aggregation lives laptop-side, where the cores are (sandbox-minimalism doctrine: the sandbox collects append-only and computes nothing).
Rollup outputs (files first — the status-quo formats are proven; a stats DB only if query pain actually appears):
~/logs/sysstat/sa-<day>(unchanged) + per-PID daily files beside them; retention 14 days to match.- The boot/stats series:
~/logs/init.jsonlpromoted to canonical — one record per boot (duration, store/profile counts, disk, latency samples), read by sweeps instead of re-probed. - Trip aggregates + per-lane proof table: computed laptop-side on a slow cadence from trips.jsonl (via
fleet get, Design: fleet proxy get/put — file transfer over the agent channel #5759) and agent.tasks; consumed by the watchdog, metrics passes, and Incident record 2026-11-04: afternoon responsiveness loss — sandbox wedged after post-boot herd (daemon restart, 88-min sampler silence); laptop path exonerated 21:00–21:28 UTC #5701-class forensics. - Consumers read rollups; nobody probes live. A consumer that needs a number nobody collects files it as a rollup addition — never as a new probe in a job's prelude.
What remains of the boot-init job — explicitly
- Append one boot record to the stats series, from caches and collector state (sub-second; no live probes).
- Invoke
fleet initprewarm: proxy up, tunnel pool warm. - Exit.
The 900-second services-verify, the per-service installers, the nix/symlink restores, and the toolchain checks are gone from init — the installers/restores land in the #5770 generic restorer (whose M3 recycle-drill acceptance also absorbs the boot-init cron's quota-recorder reinstall side-role), and verification becomes a rollup read. When the restorer lands, ~/tools/init.sh can be deleted outright: fleet init — stats append + prewarm — is the whole job, exactly the two purposes Drew named.
Interplay with the metrics enumeration's absent-metrics list
- Init-duration series: the raw series already exists (
init.jsonl); what was absent is a consumer/rollup — this design names it canonical (the enumeration's "only the latest boot is kept" is true ofinit-status, not of the JSONL). - Per-lane shift proof: becomes a rollup output (item Bump build/buck/preludes/buck2-prelude from
3a02ea3to3e4d218#8), retiring the watchdog's live joins. - Trip failure taxonomy: the aggregates half lives here; the four-timestamp instrumentation itself is Trip logging v2: four timestamps per trip (sent / received / duration / returned) for per-leg attribution #5702.
Links: #5701 (incident that exposed the sampler gaps), #5702 (trip log v2 — its aggregates belong to this rollup), #5775 (standing-load rulings; quota-recorder cadence stays Drew's call there), #5687 (fleet init location semantics), #5770 (generic restorer — receives everything init sheds).
Acceptance criteria
- The design inventories what
fleet initand the boot-init job do today, and classifies every duty as keep-stats, keep-prewarm, delete, or re-home (with the new home named). - It states the end-state contents of the boot-init job explicitly, and shows the result passes Drew's two-purpose test (stats + prewarm only).
- The telemetry rollup design names one pipeline (sysstat base + fleet extra stats) with its outputs and consumers.
- Implementation follow-ups are identified as separate issues/tasks — the design itself implements nothing.
Metadata
Metadata
Assignees
Labels
Medium: expected under 1 day of workA design proposal (design-architect): multi-step, phased, decision-bearingIntake review done: acceptance criteria, links, and normalized labels addedOrdinary priority (the default)Has a queued task in the Supabase agent queueNot yet triaged; needs a first lookQueue packet kind: verify, reproduce, and route this item
Activity
developing-today commentedon Nov 5, 2026
CODE-1602
dezren39 commentedon Nov 5, 2026
Intake review
Related: Framed by Drew 2026-11-04 20:48 CST; follows WS-C's init fix (75 s → 9 s). The monitor-digest/init-ensure path (
bash ~/tools/init-ensure.shruns at every job start) is a consumer of whatever this decides.Still-live check: searched merged PRs referencing this issue at 2026-11-05T04:31:57Z — none found; the issue stands as filed.
Notes / risks: Init runs at every job start fleet-wide, so a rollup that adds latency taxes every run. Deleting a duty that something silently depends on (prewarm) would surface as cold-start failures elsewhere.
Suggested approach: Start from the measured init duty list (WS-C timings), interview the consumers of each output, then draft the end-state and the migration order.
Acceptance criteria were appended to the issue body and labels normalised by the fleet monitor's intake pass.
dezren39 commentedon Nov 5, 2026
Retention/export audit (2026-11-04 ~23:20 CST, prompted by Drew's question on this issue's rollup design):
Current state, verified by direct reads:
scripts/sysstat/sysstat-collector.sh) writes sar data to~/logs/sysstat/sa-YYYY-MM-DD— survives sandbox recycles (under /home/hatch) BUT retention is a 14-day prune (find -mtime +14 -deletein the collector). Not exported anywhere;~/logsis not tracked by the hatch-personal repo, so the git backup does not cover it either.~/logs/resource, daily files Oct 26 → today (~72 MB), recycle-safe, no export path found.~/logs/init.jsonl— recycle-safe, no export.~/.local/state/laptop-agent/trips.jsonl(2,982 lines) on the laptop disk; no export.GAP for the rollup design to close: the rollup outputs have no specified durable home or retention. Drew (2026-11-04): rolled-up aggregates are fine as the long-term form — but they need a store that (a) survives sandbox recycles, (b) is exported/backed up (candidates: Supabase muse-dev like quota-recorder/automation.events, Turso, or laptop disk covered by laptop backup), and (c) has an explicit retention (indefinite for rollups is acceptable and cheap at aggregate size). Raw retention stays short (14d sar is fine) once rollups land somewhere durable.
dezren39 commentedon Nov 5, 2026
CORRECTION to the audit comment above (2026-11-04 ~23:38 CST) — the git/export claims there were wrong; a deeper audit plus direct re-verification establishes:
~/logsis a SYMLINK into the hatch-personal repo (~/hatch-personal/logs). Inside that repo,logs/init.jsonland thelogs/sysstat/sa-*daily files ARE git-tracked → pushed to GitHub (dezren39/hatch-personal). The earlier check read the outer repo through the symlinked path and got a false "untracked".sandbox-home-backup→ R2 (retention verified in the script: keep-daily 14 / keep-weekly 8 / keep-monthly 12).logs/resource/is gitignored in hatch-personal — R2 backup only.89 remaining items
dezren39 commentedon Nov 6, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 6, 2026
🤖 Agent note · handed off