Skip to content

Live E2E: nothing pins the key gates, so a step that maps the secret can run ungated #811

Closed
Birfy/FlyBot
#32

Description

@Birfy

What happened

Reviewer2's negative control on #30 (b6cdfbd1) found the hole: deleting if: from Run the four checks in .github/workflows/live-e2e.yml leaves all four new tests green. The invariant "a keyless run touches nothing that needs the key" is real in the workflow and is checked by nobody but the person reading the diff.

#30's own tests cover the preflight script and the skip path. They do not read the workflow text, so a later edit — dropping an if:, or adding a step that maps the secret — changes what a keyless run does and CI says nothing. The failure direction today is a red step rather than a silent pass, which is why this is not urgent; it is still an invariant with no test behind it.

The inventory to pin, at 6f9b60c7 (live-e2e.yml blob 083bf76d)

Step Lines env maps secrets. if:
Preflight (a missing key skips every step, and says so) — id: preflight 88–92 yes none, by design — it is the decider
Start the real-model bench 125–131 yes steps.preflight.outputs.key == 'true' (126)
Run the four checks — id: run 132–142 yes steps.preflight.outputs.key == 'true' (133)

The six key-consuming steps that do not map the secret themselves are gated the same way: npm ci (core) (94), Install the pinned DSH runtime (99), Clone DeepSeek Harness at the pinned tag (107), Start the usage tap (114), Start the real-model bench (126), Run the four checks (133), plus Stop the bench and the tap (144).

Acceptance

A test in .github/scripts/test-workflow.test.mjs — the file that already asserts workflow text — reading .github/workflows/live-e2e.yml as text and asserting, per step block (split on step boundaries; an if: in a neighbouring step must not satisfy this one):

  1. Every step whose env: maps secrets. carries a non-empty if:, except the step whose id: is preflight.
  2. Each such if: names steps.preflight.outputs.key.
  3. The seven key-consuming steps listed above carry the same if: (or, for Stop the bench and the tap, always() && steps.preflight.outputs.key == 'true').
  4. The preflight step itself is still unconditional — a regexp that requires an if: on every secret-bearing step must exempt it by id, never by the display name.

The test must be shown failing on all five of these edits, one at a time, in the pull request:

  • delete if: from Run the four checks → red, naming that step;
  • delete if: from Start the real-model bench → red;
  • delete if: from npm ci (core) → red;
  • add a step with env: { DEEPSEEK_API_KEY: ${{ secrets.ANYTHING }} } and no if: → red;
  • put any if: on the preflight step → red.

Kill conditions

  • The test passes while any one of those five edits is applied.
  • The exemption is keyed on the step's display name, or on being the first step, instead of id: preflight.
  • A YAML parser is added as a dependency, or the assertion is a single regexp over the whole file rather than over each step block.
  • live-e2e.yml itself is changed in the same pull request. This pins what exists; a workflow change is its own issue.

Out of scope

Adding a real DEEPSEEK_API_KEY or making a keyless run count as verification (that is #794 and it stays needs-human); the preflight script's own behaviour (.github/scripts/live-e2e-preflight.test.mjs already covers it); the four E2E checks; any other workflow.

Activity

  1. added
    bugSomething isn't working
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    area:automationScheduled tasks and automation
    on Oct 23, 2026
  2. Birfy commented on Oct 23, 2026

    @Birfy
    CollaboratorAuthor

    Implementation notes, read from 6f9b60c7 so nobody has to re-derive them:

    • Where it runs. docs/testing-layers.json has a row id: scripts whose paths is the glob .github/scripts/*.test.mjs, and CI runs exactly that glob at .github/workflows/test.yml:113, also inside npm run verify. A new sibling file is therefore picked up with no manifest edit and no workflow change — the manifest row already covers it, and testing-layers --check does not need a new row.
    • If testing-layers --check still reports a count mismatch after the file is added, take its instruction to regenerate docs/testing-layers.md; the scripts row itself is unchanged.
    • Still required: a changelog.d/ fragment (changelog check), and the three assertions must be per step block — a global regexp over the file would let a neighbouring step's if: satisfy this one, which is the same class of hole this issue exists to close.
    • The exemption stays keyed on id: preflight, never on the step's display name. Preflight must keep running unconditionally; if it ever gets an if:, a missing key produces no warning and no skip, which is worse than the ungated step this pins.
  3. Birfy commented on Oct 23, 2026

    @Birfy
    CollaboratorAuthor

    Branch: test/811-live-e2e-key-gates on Birfy/FlyBot, pull request Birfy#32 (base main 6f9b60c7, tip 7b541a12). One file, .github/scripts/test-workflow.test.mjs +49; live-e2e.yml is untouched.

    All five edits from the acceptance were applied one at a time and each reds the case that names the step: Run the four checks 2 red, Start the real-model bench 2 red, npm ci (core) 1 red (lost its gate), a rogue secret-mapping step 1 red ("Rogue step" maps a secret without an if:), and an if: on the preflight 1 red. Restored: 6/6, and live-e2e.yml compares byte-identical to 6f9b60c7. No YAML parser, no whole-file regexp, exemption by id: preflight.

  4. added
    agent-prPull request opened by an agent
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    and removed
    agent-readyMaintainer-approved spec: an agent may take it. Only maintainers set this; never in a template
    on Oct 23, 2026
  5. removed
    agent-workingAn agent has claimed this issue (the claim comment names the branch)
    agent-prPull request opened by an agent
    on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:automationScheduled tasks and automationbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions