Repository navigation
Description
What happened
Reviewer2's negative control on #30 (b6cdfbd1) found the hole: deleting if: from Run the four checks in .github/workflows/live-e2e.yml leaves all four new tests green. The invariant "a keyless run touches nothing that needs the key" is real in the workflow and is checked by nobody but the person reading the diff.
#30's own tests cover the preflight script and the skip path. They do not read the workflow text, so a later edit — dropping an if:, or adding a step that maps the secret — changes what a keyless run does and CI says nothing. The failure direction today is a red step rather than a silent pass, which is why this is not urgent; it is still an invariant with no test behind it.
The inventory to pin, at 6f9b60c7 (live-e2e.yml blob 083bf76d)
| Step | Lines | env maps secrets. |
if: |
|---|---|---|---|
Preflight (a missing key skips every step, and says so) — id: preflight |
88–92 | yes | none, by design — it is the decider |
Start the real-model bench |
125–131 | yes | steps.preflight.outputs.key == 'true' (126) |
Run the four checks — id: run |
132–142 | yes | steps.preflight.outputs.key == 'true' (133) |
The six key-consuming steps that do not map the secret themselves are gated the same way: npm ci (core) (94), Install the pinned DSH runtime (99), Clone DeepSeek Harness at the pinned tag (107), Start the usage tap (114), Start the real-model bench (126), Run the four checks (133), plus Stop the bench and the tap (144).
Acceptance
A test in .github/scripts/test-workflow.test.mjs — the file that already asserts workflow text — reading .github/workflows/live-e2e.yml as text and asserting, per step block (split on step boundaries; an if: in a neighbouring step must not satisfy this one):
- Every step whose
env:mapssecrets.carries a non-emptyif:, except the step whoseid:ispreflight. - Each such
if:namessteps.preflight.outputs.key. - The seven key-consuming steps listed above carry the same
if:(or, forStop the bench and the tap,always() && steps.preflight.outputs.key == 'true'). - The preflight step itself is still unconditional — a regexp that requires an
if:on every secret-bearing step must exempt it byid, never by the display name.
The test must be shown failing on all five of these edits, one at a time, in the pull request:
- delete
if:fromRun the four checks→ red, naming that step; - delete
if:fromStart the real-model bench→ red; - delete
if:fromnpm ci (core)→ red; - add a step with
env: { DEEPSEEK_API_KEY: ${{ secrets.ANYTHING }} }and noif:→ red; - put any
if:on thepreflightstep → red.
Kill conditions
- The test passes while any one of those five edits is applied.
- The exemption is keyed on the step's display name, or on being the first step, instead of
id: preflight. - A YAML parser is added as a dependency, or the assertion is a single regexp over the whole file rather than over each step block.
live-e2e.ymlitself is changed in the same pull request. This pins what exists; a workflow change is its own issue.
Out of scope
Adding a real DEEPSEEK_API_KEY or making a keyless run count as verification (that is #794 and it stays needs-human); the preflight script's own behaviour (.github/scripts/live-e2e-preflight.test.mjs already covers it); the four E2E checks; any other workflow.
Activity
Birfy commentedon Oct 23, 2026
Implementation notes, read from
6f9b60c7so nobody has to re-derive them:docs/testing-layers.jsonhas a rowid: scriptswhosepathsis the glob.github/scripts/*.test.mjs, and CI runs exactly that glob at.github/workflows/test.yml:113, also insidenpm run verify. A new sibling file is therefore picked up with no manifest edit and no workflow change — the manifest row already covers it, andtesting-layers --checkdoes not need a new row.testing-layers --checkstill reports a count mismatch after the file is added, take its instruction to regeneratedocs/testing-layers.md; thescriptsrow itself is unchanged.changelog.d/fragment (changelog check), and the three assertions must be per step block — a global regexp over the file would let a neighbouring step'sif:satisfy this one, which is the same class of hole this issue exists to close.id: preflight, never on the step's display name.Preflightmust keep running unconditionally; if it ever gets anif:, a missing key produces no warning and no skip, which is worse than the ungated step this pins.Birfy commentedon Oct 23, 2026
Branch:
test/811-live-e2e-key-gatesonBirfy/FlyBot, pull request Birfy#32 (basemain6f9b60c7, tip7b541a12). One file,.github/scripts/test-workflow.test.mjs+49;live-e2e.ymlis untouched.All five edits from the acceptance were applied one at a time and each reds the case that names the step:
Run the four checks2 red,Start the real-model bench2 red,npm ci (core)1 red (lost its gate), a rogue secret-mapping step 1 red ("Rogue step" maps a secret without an if:), and anif:on the preflight 1 red. Restored: 6/6, andlive-e2e.ymlcompares byte-identical to6f9b60c7. No YAML parser, no whole-file regexp, exemption byid: preflight.test(ci): pin the live-e2e key gates so a deleted if: cannot pass (Th…
Read a secret from every carrier a step has, not only a step-level en…