mcpx testing: a CI time budget in the final job, and the over-budget runbook #1995

Relocated from dezren39/darwin-nix#140 on 2026-10-28 (placement audit): mcpx's single source lives in developing-today/code (pkgs/mcpx); re-home tracked at #309. Original body below; issue references inside it are historical (dezren39/nix numbering).

Parent: #95

Summary

Add a CI time budget: a hardcoded default (overridable per run), a final job after the matrix that measures how long the run took and which part was slow, and a written runbook for what to do when the budget is exceeded. Recommendation: report several measures, gate on the longest leg, warn rather than fail on PRs, and track sustained overruns on main. The final job doubles as the single required check (ci-ok) defined in #135, so the budget costs no extra job start.

Original notes

• testing
  • importantly, as we add more testing agents should not run every single test basically ever. below documents a rough idea of a ci job design/policy, the main priority is being short as possible.
  • make a comprehensive github action ci.yaml workflow (or test.yaml) that is as fast as possible,
    • continue until the jobs can't get much faster considering setup/teardown and then set a ci testing budget which is a hardcoded default input. after the test matrix have a time budget job when subtracts github workflow request time from now. (this catches if the jobs are impacted by matrix size, could lessen by taking initial job end or matrix start time.)

• testing
  • make a comprehensive github action ci.yaml workflow (or test.yaml) that is as fast as possible,
    • repeat this process whenever testing exceeds some time budget. make setup faster, rebalance alpha splits, split groups of tests on the longest shard. maybe flag that never tears down because just killing the box after. anything downloaded should be cached, shallow clone a partial path to mcpx dir, if there's unexplainable job start delays, try to keep all job times balanced, 10 ultrafast ones can give you quick signals but may put pressure on runner infra and cause the longest jobs to last a little longer. all that matters is how long the longest jobs take. job runners have 2 or 4 cores, consider if theres lengthy tests which are '1' test but re-exec N times 'in parallel': they will only run 1-3 at a time and so 30 cases could take up to 10 tests worth of time even in parallel and may need to be examined. we're using go so im not too worried about time budget but its best to get this early. main focus is strong tests as each step of the lifecycle.

Current state

• No budget and no timing report exist. Wall times of the last 20 mcpx.yml runs: 3m24s-4m30s; the last 10: 3m57s-4m30s (a ~15 % spread on an unchanged workload -- the budget must have headroom or it flakes). The runs since (8001f23's merge, Phase 5: tag event bodies must travel with the event #182's PR and merge at 05c78b2, Renovate PR Phase 5: compaction for tag clears #184): 4m03s-4m15s, with e2e now at 209 tests. The gap between summed step time (229 s on 36673523537) and reported run time (237-258 s on the eight runs before it) is 8-29 s of queue and runner provisioning -- the quantity the notes worry about contaminating the number (from ci(repo): stabilize formatting, Go tidy, and id Nix checks #167).
• There is no run-start timestamp in any expression context. The github context has run_id, run_number, run_attempt, workflow_sha, ... but no time (contexts reference). The times come from the REST API with permissions: actions: read:
  • GET /repos/{owner}/{repo}/actions/runs/{run_id} -> created_at, run_started_at, updated_at. run_started_at is the start of the current attempt; created_at is the first attempt's. They are equal on attempt 1 (both 2026-09-30T05:27:57Z for run 36673523537) and differ on a re-run -- use run_started_at, or a re-run is measured from the original push.
  • GET /repos/{owner}/{repo}/actions/runs/{run_id}/attempts/{attempt}/jobs -> per job created_at (queued), started_at, completed_at, and per step started_at/completed_at. Verified on run 36673523537: job created 05:27:58, started 05:28:00, completed 05:31:53; step times match the log.
• The notes' caveat is real: "now minus run start" includes the budget job's own queue and startup, init, and any queueing caused by matrix size (the Free plan runs 20 standard jobs at once account-wide, limits).

Approaches

What to measure

• A -- only now - run_started_at (the notes' first idea). Simple, and it is what a human waiting on the PR feels. But it mixes queueing, init, the budget job's own start (~5 s + queue), and the real test time, so when it goes over, it does not say why. Least recommended as the only number.
• B -- measure from init completion / matrix start (the notes' alternative). max(completed_at of matrix legs) - init.completed_at. Removes init and the final job's own start; still includes queue delay for legs. Of the notes' two spellings, prefer init end over "matrix start": the gap between init finishing and the legs starting is the matrix-size effect the notes want caught, and measuring from the first leg's start deletes it (from ci(repo): stabilize formatting, Go tidy, and id Nix checks #167). init can also publish its own end time as a job output (date +%s into $GITHUB_OUTPUT), so this number needs no API call.
• C -- report all of them, gate on the longest leg (recommended).

Gate on longest leg (controllable by us); report wall prominently; show queue so an over-budget wall with a normal longest leg is visibly GitHub's queue, not our tests. All go into the job summary as one table, plus a timings.json artifact merged from the legs' go test -json output (the input #139's splitter reads).

The budget input

"Hardcoded default input" = a literal in the workflow, overridable when dispatched by hand:

on:
  workflow_dispatch:
    inputs:
      budget_seconds: { type: number, default: 0 }
env:
  # Longest-leg budget. Changing it is a reviewed edit with a reason; see the runbook.
  BUDGET_SECONDS: ${{ inputs.budget_seconds > 0 && inputs.budget_seconds || 120 }}

inputs is empty on push/pull_request, so those use the literal. A repository variable (vars.MCPX_CI_BUDGET) would let someone change the budget without a commit -- rejected, because the notes ask for a hardcoded default and a budget that changes silently is not a budget. A committed file (e.g. .github/ci-budget.json, { "longest_leg_seconds": ..., "wall_seconds": ... }, read by init and passed down as an output) is the other reviewable home: a budget change becomes a one-line data diff beside the timing file #139 reads, and the dispatch input still overrides it (from #167). Either is fine; the literal is one file fewer.

The number: set it after #143's baseline and #139's first sharding, at roughly the p90 longest leg of ~10 main runs plus 15 % (the observed spread), then ratchet down as the runbook wins time. 120 above is a placeholder.

What happens over budget

• A -- fail the run. With a ~15 % run-to-run spread, a tight budget fails healthy PRs, people re-run until green, and the budget becomes noise that costs more runner time than it saves. Not recommended.
• B -- warn only (::warning:: annotation + summary). Cheap and honest, but a warning on every PR is ignored within a week.
• C -- warn per run; act on sustained overruns; fail only on gross overruns (recommended).
  • Every run: summary table + a ::warning:: if the longest leg exceeds the budget.
  • On main only: if the median longest leg of the last 5 main runs (from the API) exceeds the budget, open or update one issue "mcpx CI over budget" (label testing) with the table and a link to the runbook. permissions: issues: write on this job, main only.
  • Fail only above 2x the budget: that is a hang or an accidental serialisation, not variance, and it should block.

Run the final job on ubuntu-slim (1 vCPU container, meant for short automation jobs; runner reference) -- it only needs gh/curl and jq. Whether ubuntu-slim starts faster or slower than ubuntu-latest is unmeasured; check in #143.

Runbook (when the budget is exceeded)

Lives as a comment block at the top of mcpx-test.yml and is linked from the summary (a longer version can go wherever #99 puts the testing policy doc). Cheapest first; stop as soon as the budget is met:

1. Read the table. Which measure is over?
   • queue high, longest leg normal -> runner pressure. Merge the smallest legs (lint legs first), or drop a shard. Do not add shards.
   • fixed cost high -> setup. Go to 2.
   • a test leg long -> go to 3.
   • the nix leg longest -> see Update Rust crate tokio to v1.38.2 [SECURITY] #135 decision 7 (store cache; question whether checkPhase needs to re-run units CI already ran).
2. Make setup faster. Look for cache misses in the log (until Phase 5: tag event bodies must travel with the event #182 every run carried Restore cache failed: Dependencies file is not found; since Phase 5: tag event bodies must travel with the event #182 a miss reads Cache is not found in the setup-go step); drop setup-* steps a leg does not use; ship the compiled e2e.test and binaries instead of building per leg.
3. Rebalance. With the Update dependency hyperscript.org to v0.9.93 #139 splitter this is automatic; check the timing file is fresh. If using hand ranges (not recommended), move the boundaries.
4. Split the longest shard. Lower target_leg_seconds / raise N -- but only while the saving per extra leg exceeds the fixed cost per leg (Update dependency hyperscript.org to v0.9.93 #139's stop rule).
5. Make the slowest tests faster. Top 10 from timings.json: fixed time.Sleep waits, fan-out tests on too few cores, anything that starts a runtime it does not need. Update dependency @playwright/test to v1.64.1 #79's CLI speedups apply to every e2e test.
6. Parallelise within a leg (t.Parallel in e2e). chore(deps): bump build/buck/preludes/buck2-prelude from d956689 to e7457f6 #52 no longer blocks it (fixed in Phase 5: tag event bodies must travel with the event #182); Update dependency htmx.org to v1.9.12 #138's environment allowlist comes first (see Update dependency hyperscript.org to v0.9.93 #139).
7. Raise the budget only as a reviewed change with the table attached and the reason in the commit message.

Repeat whenever the main-branch issue opens.

Open questions

1. Budget on longest leg only, or also a separate (looser) wall budget?
2. Where does the long-form runbook live -- workflow comment only, or the testing policy doc from Bump unsafe-libyaml from 0.2.8 to 0.2.10 in /sources/pavex/examples/realworld #99?
3. Should the over-budget issue be one rolling issue or a comment on Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sources/launcher #95?
4. ubuntu-slim vs ubuntu-latest for the final job: start latency?
5. Are timings from re-run attempts excluded from the median? (They are usually warm and faster.) Record github.run_attempt beside every measurement so the history stays interpretable.
6. Should a leg's setup have its own threshold? A setup regression is invisible in wall clock while the matrix absorbs it, and setup is the floor on the useful shard count (Update dependency hyperscript.org to v0.9.93 #139's stop rule) (from ci(repo): stabilize formatting, Go tidy, and id Nix checks #167).
7. Bump golang.org/x/crypto from 0.14.0 to 0.17.0 in /sources/chat #97's environment legs (container, remote, macOS) are legitimately slower: one global budget, or one per matrix dimension (from ci(repo): stabilize formatting, Go tidy, and id Nix checks #167)?

Ease · Impact · Related

Ease S (the job) + S (runbook) · Impact med · Depends on / related: #95, #135 (the final ci-ok job this lives in), #139 (consumes timings.json; stop rule), #143 (baseline that sets the number), #138 (before t.Parallel), #52 (closed, fixed in #182), #79, #97 / #134 (environment legs will likely be the longest -- macOS runners are 3 vCPU and capped at 5 concurrent), #130 (its PR job is inside the budget, its nightly run is not), #99.

Security/auth deferred; see tracker.

Acceptance criteria

• The final CI job enforces/reports the stated time budget and the over-budget runbook is recorded where the body specifies.
• The work described in the Summary/Scope above is delivered via a branch + PR (no direct push to main); verification runs in GitHub Actions CI, not on the sandbox.
• Evidence (CI links, timings, or probe output as applicable) is recorded on this issue before it is closed.