stall guard: add a process-level liveness probe as a second signal beside output flushes — with the naive-CPU inversion hazard designed out

Follow-up to #11808 (which stays open until its own PR lands; this card is deliberately separate and is NOT addressed there). That fix pins --log-order=stream under every guard-wrapped turbo and makes the guard refuse a turbo invocation without it, so the guard's output-flush instrument observes real liveness again. This card records the second, more durable correction that #11808's triage graded as in-scope but which was deferred on measurement: the smaller change restores the behaviour the guard was written against, and the probe carries a failure mode that needs its own design pass rather than a rider.

Why a second signal is still worth having

The guard's only instrument is output flushes. Any future buffering layer between the suite and the guard — a vitest reporter change, a pnpm output-mode change, a new runner that groups logs — silently re-creates the #11808 trap: a healthy long task reads as frozen and is killed deterministically, and the resulting red reproduces on rerun, which the standard flake triage then confirms as "the diff". The refusal shipped with #11808 closes exactly one spelling of that class (turbo without --log-order=stream); a process-level liveness signal would close the class itself.

The raw signal is already in hand: on a declared stall the guard's forensic sampler (sampleGroup() in scripts/run-with-stall-guard.mjs) reads /proc state + cpuTicks for every process in the wrapped group and classifies each as ON-CPU vs idle, and the SIGUSR2 harvest distinguishes a live event loop from a blocked one. Today all of that runs only after the verdict, as forensics. The probe would move a version of it before the verdict.

The hazard that makes this a design task, not a one-liner (do not skip this)

A naive probe — "the group is burning CPU, therefore it is alive, therefore do not kill" — inverts the defect instead of fixing it. A sync-spinning hang (infinite loop, GC thrash: the guard self-test's case 5, and half of the #4250 population) has CPU ticks forever, so cpu-alive-means-no-kill converts "kills healthy suites" into "never fires on spin hangs", which is strictly worse: the job then sits in_progress until the job timeout, which is the exact state the guard was built to abolish — and no green run can distinguish that regression from success.

Any acceptable design must therefore state, in the PR body, what a genuinely frozen suite looks like to the probe, and must extend --self-test so that:

the idle hang (case 4) still fires,
the sync-spinning hang (case 5) still fires,
a healthy silent-but-working shape (the thing the probe exists to protect) does NOT fire.

If those three cannot all hold simultaneously, the probe is wrong, not the cases.

Directions that survive the hazard at first look (to be measured, not assumed): treating CPU as a confirming signal only ever in the kill-delay direction bounded by a hard cap far above --stall-minutes (so a spin hang is still killed, later, with a distinct verdict word); or probing task-level progress (e.g. vitest's per-file lifecycle via a reporter side-channel file the guard watches) instead of CPU at all, which measures the quantity the guard actually cares about.

Scope

scripts/run-with-stall-guard.mjs (+ its --self-test), possibly a small reporter shim if the task-progress direction wins.
No CI wiring change expected beyond flags the guard itself documents.
Independent of --stall-minutes budgets; ⛔ raising the budget stays a gate weakening on the maintainer's floor (per CI: run-with-stall-guard kills a HEALTHY solo tail task — turbo grouped log order flushes nothing while one long task runs, and cli:test now outgrows the 10-minute silence budget #11808's triage grading) and is not part of this card.

Refs: #11808 (mechanism, occurrences, and the measured evidence) · #4250 (the real-stall family) · scripts/run-with-stall-guard.mjs header.

Generated by Claude Code

Triage (triage seat, session session_01PCuhUdqwK7zkkxgtJHswTc): lands in scripts/run-with-stall-guard.mjs (+ --self-test, possibly a reporter shim) → domain:devx, pm:queue, type Task. Rationale: a design task inside CI tooling with the hazard already designed out in the card (three self-test cases that must hold simultaneously); no product-semantics or gate-weakening question — the card itself fences off budget raising as a maintainer floor.

Serial-constraint note for the devx seat: same file as #11808 (in flight, os-steve). Hard same-file serial applies — this card waits until #11808's PR lands, and its premise must be re-read on the merged ref (the --log-order=stream refusal is this card's stated baseline). Fold-or-serial answer: serial, not fold — #11808 is a repair, this is a design pass; five-gate test fails on "same defect shape, same fix".

Size/model suggestion: M (design judgment in the probe semantics), mode:subagent.

Generated by Claude Code

Claim: PM dispatch (devx seat)

Session: session_015ahemw8RcTgqtxrj15PEZx
Branch: claude/issue-11855-stall-guard-liveness-probe

The serial constraint triage set is RELEASED. Triage wrote: "Hard same-file serial applies — this card waits until #11808's PR lands, and its premise must be re-read on the merged ref." #11808's PR is #11868, and it has landed: e75e34381 fix(ci): stall guard reads real liveness — --log-order=stream under every guard-wrapped turbo, guard refuses grouped, shard timings re-measured (#11868). Verified by content on origin/main, not by sha — turboLogOrderViolation is live at scripts/run-with-stall-guard.mjs:155, the refusal fires at :167, and the --self-test carries both directions of the precondition at :587–:614. Positive control: a token that cannot exist counted 0 in the same read.

So the premise re-read triage required is now possible, and it is required, not optional: read the guard on the merged ref before designing, because #11868 changed the very instrument this card is adding a second signal beside.

Tier: opus. Derived live, not recalled — node scripts/pm/dispatch-gates.mjs --tier scripts/run-with-stall-guard.mjs reports "no path-derived mandate: the surface hits none of the 3 declared glob(s)", so the floor is sonnet and the tier is this seat's judgment. opus because the load-bearing work is the probe's semantics, not the edit.

Generated by Claude Code

{
  "issue": 11855,
  "status": "done",
  "branch": "claude/issue-11855-stall-guard-liveness-probe",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/11915",
  "premise_still_valid": true,
  "summary": "Re-read the guard on the merged ref (#11868 content verified live: turboLogOrderViolation, the refusal, both --self-test directions) and confirmed both Zone 2 assumptions before building: sampleGroup() does read /proc state + cpuTicks per process, and self-test cases 4/5 are numbered as the card says. Then measured the two offered directions instead of inheriting them. Direction (a), CPU as a confirming signal, LOST on measurement and lost harder than the card feared: over a 2.5s sample the idle hang moved 0 CPU ticks, the sync-spinning hang 251 and the GC-thrash hang 272, while the healthy silent-but-working shape the probe exists to protect moved 1 — it is I/O-bound, it is busy writing. CPU does not merely fail to separate those shapes, it separates them BACKWARDS, so a CPU-confirming probe would have protected the hangs and killed the healthy suite. Direction (b), a vitest reporter side-channel, measures the right quantity but is not reachable at proportionate cost: this repo has no shared vitest preset (31 packages, 14 own vitest.config.ts, every test script a bare `vitest run`, guard wraps `pnpm turbo run test`), so it is a 31-file rollout through every package's test infrastructure. Shipped instead the signal the measurement selected: source-side write bytes per process, read from /proc/[pid]/io wchar, which separated all four shapes correctly (healthy = 25,542 bytes, all three hangs = 0). It is the same quantity the guard already trusts, sampled before any buffering layer can hide it, with zero producer wiring and no code injected into any test process. The hazard is contained with the card's own structure applied to the right signal: the probe is confirming in ONE direction, it can only DELAY a kill, the deferral is announced loudly (a live suite hidden by a buffering layer is a bug to fix, not tolerate), and it is bounded by --stall-cap-minutes after which the group dies under a distinct STALL-CAP verdict. --stall-minutes is untouched. A genuinely frozen suite, to this probe, is one where ZERO bytes were written by ANY process in the group for the whole stall window — frozen means produced nothing, not did nothing, and explicitly not 'no CPU'.",
  "tests": "All at HEAD=fbc723061 (union run after the final commit). `pnpm check:stall-guard` -> its own verdict line `✓ 41 case(s) passed — the guard fires, classifies and tears down.` (27 assertions on origin/main, +14 here). The three required conditions hold SIMULTANEOUSLY in that run: `✓ idle hang is declared a stall` + `✓ idle hang is not rescued by the liveness probe (plain verdict, no deferral)`; `✓ sync-spinning hang is declared a stall` + `✓ sync-spinning hang is NOT rescued by the probe — burning CPU is not liveness`; `✓ a healthy silent-but-working run is NOT killed` + `✓ ...and the deferral is announced, not silent`, with the positive control `✓ positive control: the same run IS killed with --no-liveness-probe` — without that control a probe that did nothing would score identically. Plus `✓ a hang that keeps writing is still killed` / `✓ ...at the cap, under its own distinct STALL-CAP verdict`. ABLATION, two mutations, each proved on disk in BOTH directions by anchored grep -cF (injected present=1, removed absent=0), each node --check'd so a crash-red cannot pass as an assertion-red, each restored under `trap restore EXIT INT TERM`, restored tree re-verified pristine (both anchors back to 1, parses): M1 `sourceAlive` forced false -> predicted the healthy case turns red, observed 5 red incl. `✗ a healthy silent-but-working run is NOT killed — exit 75`. M2 `capped` forced false -> predicted the writing hang is never killed, observed 3 red incl. `✗ a writing hang: the guard still exits on its own — the guard never exited — the probe inverted the defect` and `exit null`; M2 reproduces the card's inversion on demand, which is the evidence that the cap is load-bearing. GATES: family derived live via `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack` (8 families, no hand-written list), all run at fbc723061 with exit codes captured before any pipe — check:stall-guard, check:agent-test-spelling, check:cross-package-test-inputs (`OK: 16 package(s) read outside themselves, all declared`), check:entry-guard (`✓ 154 scripts/ file(s) — every entry guard goes through invoked-as.mjs`), check:parse-guard, check:pnpm-filter-targets (`✓ 135/168 --filter occurrence(s) across 26 file(s) resolve`), check-ci-filter-parity (`OK: all 96 declared cross-package glob(s) (81 unique) are covered`), check-cross-package-test-inputs, plus check-nul-bytes (`OK (scanned 6631 text file(s))`) — all green. check-ci-filter-parity first exited 1 self-reporting `PREREQUISITE NOT MET — the dependency yaml is not installed` and `nothing was measured … this result says NOTHING about what it gates`; green after `pnpm install` in the worktree. skip-changeset applied to PR #11915 via the additive endpoint and READ BACK present.",
  "open_questions": [],
  "out_of_scope_findings": [
    "filed as #11916: nothing checks a guard-wrapped step's kill budget against its own job's `timeout-minutes` — verified no script under scripts/ reads timeout-minutes at all. Current values are all fine (windows 10/15 against timeouts 30/120, and this card's derived caps land 20<30 and 30<120), but the invariant is unenforced and a guard that cannot speak before its job dies is a silent no-op that scores identically to a working one on every green run."
  ]
}

Design note the card asked for explicitly, restated outside the JSON so it is not lost in a field: what a genuinely frozen suite looks like to this probe is zero bytes passed to write() by any process in the wrapped group for the whole stall window. Not "no CPU" — the measurement in the PR body shows CPU separates the required shapes backwards. The full reasoning, the four-shape measurement table, the reason each of the card's two directions lost, and the cap's measured ceiling (timeout-minutes: 30 against --stall-minutes 10 in every guard-wrapped job, so "far above the window" was not available) are in the PR body.

Generated by Claude Code

Navigation Menu

Repository navigation

stall guard: add a process-level liveness probe as a second signal beside output flushes — with the naive-CPU inversion hazard designed out #11855

Why a second signal is still worth having

The hazard that makes this a design task, not a one-liner (do not skip this)

Scope

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

stall guard: add a process-level liveness probe as a second signal beside output flushes — with the naive-CPU inversion hazard designed out #11855

Description

Why a second signal is still worth having

The hazard that makes this a design task, not a one-liner (do not skip this)

Scope

Activity

os-zhuang commented on Aug 24, 2026

os-steve commented on Aug 25, 2026

os-steve commented on Aug 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions