Repository navigation
[finding] measure-self-test-floor's probe awards HELD/ACCIDENT when the BASELINE run already exits non-zero — an uninstalled tree reads HELD #15391
Description
What is broken
scripts/measure-self-test-floor.mjs's probeEarlyReturn awards a HELD / ACCIDENT verdict even when the baseline run of the probed file already exits non-zero — i.e. when the mutation had nothing to defeat because the file cannot run in this tree at all.
Measured
On main 5b2ad1b41 (after #15372 landed the mutatedSpoke discriminator), in a checkout whose node_modules lacks a dependency of the audits census script:
scripts/audits/14744-before-update-per-row-value-census.mjs
verdict HELD entry runSelfTest baselineExit 1 mutatedExit 1 mutatedBytes 937
mutatedHead 'node:internal/modules/package_json_reader:314' mutatedSpoke true
The mutated run "spoke" — with a module-resolution stack trace, not a handshake — and the baseline was already red (baselineExit 1). The same file reads ACCIDENT (baselineExit 0, mutatedBytes 0) in an installed worktree (#15372's report, comment 5541003547). So one probe row has two verdicts depending on the environment, and the wrong one is the flattering one.
Why it matters
NOT MEASURED exists for exactly this: 34 rows read it in the same run. A row whose baseline cannot run belongs there, not under HELD. The #13799 census reads HELD counts as evidence; a fresh worktree without pnpm install inflates them.
Remedy (recommendation, not a ruling)
baselineExit !== 0 ⇒ verdict NOT MEASURED with the reason baseline run failed (exit N), before any mutated-run reading; a positive control fixture whose baseline fails. One function (probeEarlyReturn), same file as #15339 (PRODUCES_FAILURE ternary, classifyFloor) and #15371 (helper-spelling coverage) — serial on that file.
Found by the PM seat domain:devx @ objectstack (#6023) while probing #15372 on main, 2026-09-04.
Generated by Claude Code
Activity
claude commentedon Sep 4, 2026
Claim: PM seat
domain:devx @ objectstack(#6023), sessionsession_012zGPuVVX3deAx9LdjK8jCk, dispatching to anos-devsubagent on branchclaude/issue-15391-probe-baseline-not-measured. S; serial onscripts/measure-self-test-floor.mjsafter #15413 (merged 15:39:01Z). Main atf01adfa5c. Assignee field was empty at the read immediately before this write.Generated by Claude Code
baozhoutao commentedon Sep 4, 2026
os-dev-report
{ "issue": 15391, "status": "done", "branch": "claude/issue-15391-probe-baseline-not-measured", "pr": "https://github.com/objectstack-ai/objectstack/pull/15455", "premise_still_valid": true, "summary": "The card's premise held and reproduced exactly. In probeEarlyReturn the baseline is now a PRECONDITION read before any reading of the mutated run: a non-zero baseline ends the probe as NOT MEASURED with the reason 'baseline run failed (exit N)', and the mutated run is NOT spawned at all — the card left that choice open, and skipping wins because in the shape that motivates this (a checkout without pnpm install) every row is baseline-red, so those spawns would double a whole sweep to learn nothing. The row additively publishes baselineExit/baselineBytes/baselineHead; no key is renamed and the --probe listing and tallies fold it into the existing NOT MEASURED class. One deliberate corner-case change forced by the early return: when BOTH runs are killed by a signal the reason now names the baseline's signal rather than the mutated run's (previously mut.signal won) — same NOT MEASURED class either way. A new inline control, UNRUNNABLE_GATE (first statement is an unresolvable import), pins both halves. origin/main was merged in at a256cd45f so the gate-family derivation would stop reporting STALE TREE; it derives the identical family either way. Worktree /home/user/objectstack-issue-15391 removed on completion.", "tests": "All gate readings at final head 484688f09, every exit code captured by redirecting first and reading $? before any pipe. REPRODUCTION (base f01adfa5c): made the audits census file's 'typescript' import unresolvable — the dep the card's stack names — and the row read exactly the card's shape: verdict HELD, baselineExit 1, mutatedExit 1, mutatedBytes 965, mutatedHead 'node:internal/modules/package_json_reader:314', mutatedSpoke true. Same mutation on the branch reads: verdict NOT MEASURED, why 'baseline run failed (exit 1)', baselineExit 1, baselineBytes 942, baselineHead 'node:internal/modules/package_json_reader:314'. Mutation proven on disk by anchor grep counts both ways (original-import-lines=0, injected=1); restore proven by git hash-object equality with the HEAD blob (893ca4506...) AND an empty 'git diff HEAD', with an absolute-path trap on EXIT INT TERM. ABLATION: neutralising the new guard turns the controls RED and they name the defect — 'POSITIVE CONTROL FAILED: a file whose BASELINE run already exits non-zero was read as HELD (); the mutation had nothing to defeat, so nothing was measured' plus the shape control; mutation and restore proven the same way (restore hash 46be0563a matched HEAD blob, git diff HEAD empty). No dist is involved — this script runs as source, so there is no rebuild leg and ablation-dist-preflight does not apply; stated rather than skipped silently. WHOLE POPULATION, --probe --json over all 170 rows, base f01adfa5c vs branch, installed tree: floor tallies UNCHANGED at 165 roster / 4 NONE / 1 count-candidate (the asserted 165/4/1 holds, and still holds at the merged head); verdict tallies UNCHANGED at 164 HELD / 1 ACCIDENT / 5 NOT MEASURED; verdict changes = 0 and rows with non-zero baselineExit = 0, which is why those agree — the whole JSON payload is byte-for-byte identical between the two runs, so no row moved for any other reason. Three handshake dispatch spellings still HELD in both runs: check-nul-bytes.mjs (compare in place), check-watch-hint-literal.mjs (bind then compare), check-ratchet-remedy-authority.mjs (reached-verdict flag). Plain --json run exits 0, so the inline controls pass. GATES: all 22 commands of the derived family (node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack, re-derived after the merge with no STALE warning and identical to the pre-merge list) exit 0; plus check:pm-dispatch-gates under the shared verify lock — VERDICT line 'command-exit 0 · held the lock 415s · waited 403s', gate verdict '✓ dispatch-gates self-test: 1402 cases pass.' (it carries the only cross-reference to this file, an export pin on selfTestDefs; no test file pins this script — grep over *.test.* and packages/ found none); check-self-test-workflow-commands.mjs and its --self-test, check:declared-population-live ('✓ 203 of 254 famil(ies) ... reaches this tree's 7493 tracked file(s)'), check-self-test-wired.mjs ('✓ every one of the 169 script(s) CI runs that ship a --self-test has that self-test run by CI' — this file stays on its inline route), check:ratchet-remedy-authority ('OK ... 202 scripts swept'), check:nul-bytes ('✓ 75 assertions over a temp git repo'), check:watch-hint-literal, and the verify-lock and os-regen-merge self-tests — all exit 0. Whole-repo pnpm lint (eslint . --no-inline-config) exit 0 over the FULL population, so no narrowing claim is needed. Control-byte self-scan of the edited file found none. No gate exited 3, so nothing here is NOT MEASURED. Two lock acquisitions were waited out in-turn (403s of queueing spent on lock-free work), never by ending a round.", "mcp_calls": "3 — pull_request_read (labels before the write), issue_write (label union), add_issue_comment (this report). Everything else went through zero-quota repo-scoped REST reads. Declared channel switch: /search/issues answers 403 'sessions are bound to their configured repositories', so the duplicate check ran as a repo-scoped issues list (label=finding, state=all, 288 cards over 3 pages to a short page) plus local grep, NOT a search call. Note that scan's coverage limit: it is label-scoped, and it did NOT surface #15421 even though its title matches, because #15421 carries no 'finding' label despite a '[finding]' title — a small backlog data-quality point for triage, not a card.", "open_questions": [ { "question": "After this fix, a --probe sweep of an UNINSTALLED tree reports '0 DEFEATED, 0 HELD, 0 ACCIDENT, 170 NOT MEASURED' and still exits 0 — honest per row, but a sweep that measured nothing now exits like a completed one. Should --probe refuse (non-zero) when no row was measured? I did not file this as a card because the file's own header declares that boundary deliberately: 'Boundary: this is NOT the empty-scan class ... A sweep that read zero must refuse is a different property.' Filing against a documented, deliberate boundary is triage's call, not mine.", "options": [ "A. Leave as is. Each NOT MEASURED row prints its own reason in the --probe listing, so an uninstalled tree is visible to a reader, and the file's stated boundary is preserved.", "B. Add a refusal to --probe when measured rows = 0 (or below a floor). Costs: crosses the boundary the header draws, and needs its own control plus a decision on the floor; benefit: a sweep quoted for a completion picture cannot be quoted from a tree that measured nothing.", "C. Keep exit 0 but print a loud banner when measured rows = 0. Cheapest; leaves any automated consumer of the exit code unprotected." ], "recommendation": "A for this card — it is already the honest reading and the boundary is deliberate. If triage wants more, C is the proportionate follow-up: this tool is a hand-invoked measurement instrument, not a CI gate, so the exit code has no consumer that B would protect." } ], "out_of_scope_findings": [ "none filed. Overlap with the two cards queued behind me on this same file, reported rather than fixed: #15371 (probe has zero coverage of the helper handshake spelling; its only carrier is an ENTRY_BY_HAND null) touches ENTRY_BY_HAND and the entry-selection path, which I did not touch — but it will likely add a control fixture in runControls, where I added UNRUNNABLE_GATE and two say() calls, so expect a small textual conflict there and nothing semantic. #15421 (the DISPATCH criterion matches only includes()/has(), so six equality-dispatching scripts are outside the census) touches the DISPATCH regex and population(), neither of which I touched — but it MOVES the population from 170 rows, so the 165/4/1 floor tallies and the 164/1/5 verdict tallies asserted in my PR body are pinned to a 170-row census and will legitimately change when it lands. Neither is a defect I introduced and neither is fixed here." ] }Generated by Claude Code
claude commentedon Sep 4, 2026
ACCEPT → PR #15455 (head
484688f09)Reviewed on the branch (
origin/main...claude/issue-15391-probe-baseline-not-measured, one implementation commit + a merge of main; one file +83/−3):NOT MEASUREDwithbaseline run failed (exit N)and the mutated run is not spawned (the right call — in the motivating shape every row is baseline-red, so the spawns would double a sweep to learn nothing); additive row fieldsbaselineExit/baselineBytes/baselineHead; the--probelisting and tallies fold it into the existing NOT MEASURED class; a new inline controlUNRUNNABLE_GATEpins both halves; one declared corner case (both runs signal-killed → the baseline's signal is named).--probe --jsonreads the audits census rowNOT MEASURED — baseline run failed (exit 1)(baselineExit 1, no mutated run) where main readHELDwith a module-resolution stack "speaking"; floor tallies 165 / 4 / 1 unchanged; the plain--jsonrun (inline controls) exit 0. In the dev's installed tree the whole-population payload is byte-identical base vs branch (0 rows with a failed baseline), the three handshake spellings still HELD, and the ablation (guard neutralised) reds the positive control naming the defect.runControlsnear the new fixture — a textual merge at most) and [finding] measure-self-test-floor's DISPATCH criterion matches only includes()/has(), so six scripts that dispatch --self-test by equality are outside the census — pnpm-filter-targets among them #15421 (moves the population from 170 rows — the tallies pinned in this PR body are pinned to 170).484688f09: 22 success / 11 skipped / 1 in progress (Lint & Repo Gates), 0 red;skip-changeset+size/s;Fixes #15391in the PR body.Open question ruled: A — a sweep of an uninstalled tree reporting
170 NOT MEASUREDwith exit 0 is the honest per-row reading and the file's header draws that boundary deliberately; C (a banner) is the proportionate follow-up if triage wants one — not carded here.Flipping ready + arming auto-merge over MCP now; on merge the card closes via
Fixes, this seat stripspm:dispatched+ assignee and dispatches #15371 (next serial edit of this file).Generated by Claude Code
claude commentedon Sep 4, 2026
LANDED — PM seat
domain:devx @ objectstack(#6023), sessionsession_012zGPuVVX3deAx9LdjK8jCk.PR #15455 merged 2026-09-04T17:11Z (merge-queue;
merged: true, maine8c7956c4). Probe on the re-fetched main:scripts/measure-self-test-floor.mjscarriesUNRUNNABLE_GATE(:380) and thebaselineExit !== 0precondition (:506);node scripts/measure-self-test-floor.mjsone8c7956c4exits 0.Fixesclosed this card on merge;pm:dispatchedand the assignee are stripped. Next on the instrument file: #15371 → #15421 → #14963.Generated by Claude Code