Repository navigation
CI stall report 2026-11-09 #4136
Description
The owner reported PR testing clogged for about 10 hours on 2026-11-09. This issue only collects data: no workflow has changed. The numbers come from bun scripts/ci-report.ts (PR linked below), read-only through gh api, over the last 7 days of apps and typecheck runs. Times are Pacific.
In short
- The wait is the queue, not slow tests. Jobs waited 23,064 job-min for a runner against 11,645 job-min of work. While anything waited, 19 to 20 jobs were running: that is the account's concurrent-job ceiling. A full
appsrun is about 60 jobs, so three PRs pushed together fill the ceiling for several minutes. The waiting is bunched up (only 5% of minutes had a job waiting), and today 09:00 to 11:00 had up to 19 runs and 468 jobs waiting at once. - Every-app runs are 90% of test job-minutes. 202 of 706
appsruns tested 38+ of the 48 apps, becausepackages/,bun.lockorci-apps.tsmoved. The usual triggers werepackages/util(77 runs, 26 apps import it),packages/cli(48, 44 apps),bun.lock(40) andpackages/ui(12, 26 apps). - A third of all job time is checkout. Checkout took 2,906 job-min (32%), and checkout, setup and install together took 44%: about 18 s per job before its first test. That cost comes from the number of jobs, not from checkout depth. The test jobs are already shallow (8 s median). The
fetch-depth: 0jobs (plan, scripts, containers-plan) take 10 s, so making them shallow saves only about 120 job-min. - Main was broken, and every PR paid for it. The qCRM demo test (
apps/qcrm/test/demo.test.ts) failed on 13 PRs on 2026-11-09 between 01:28 and 02:29 (17 failures on 14 PRs across 11-05 and 11-09). Smaller cases:agent-officeagent-routes.test.tson 11-08 (4 PRs), and the 4 pinned-release tests (floorguard,hosts,power,env) on 11-08 (3 PRs). The report counts these per test and per day. "Untouched" undercounts this: a PR that changedpackages/utilcounts as touching qCRM, so the report uses "same test failing on 3+ PRs in one day" as the main-broken signal. - What a sensible cut saves, out of 11,645 job-min a week. The three overlap, so don't add them up.
- Test only the apps that import a changed package, and run every app nightly: about 2,700 job-min.
- Run the sub-minute app suites six to a job: about 2,240 job-min, and about 35 fewer queued jobs per every-app run.
- Move
skate-tech-park(6.9 job-min a run, caught 1 failure of its own in 215 runs) andtwo-fronts(4.9, caught 2 in 218) to nightly: about 2,490 job-min.codesplitter,mythic-mastersandqcrmcaught real failures (44, 12 and 17), so they stay per PR.
Proposal: scheduled suites plus an agent loop (described only, not built)
suites-nightly.yml(cron, for example 09:00 UTC, plusworkflow_dispatch): one job per moved suite, built from the samescripts/ci-apps.tsrows (aschedule: "nightly"field on the row, whichci-apps.ts testthen skips on PRs), plus every app once a day as a safety net for the import-graph scoping. Standard runners and timeouts as today (ci-guard.ts).- On failure, one issue per failing suite. A last step (with
issues: writeon that job only) runs a small script that looks for an open issue titledCI nightly: <suite> failing. If none exists, it opens one with thecurrent-needlabel, the run link, the failing test names (the samefailsOfparseci-report.tsuses) and the first bad commit between the last green nightly and this one. If one is already open, it adds a comment instead. When the suite passes again, the step comments "green again" and closes the issue. So at most one issue per suite, never one per run. - The agent loop. A recurring office job, shaped like
bun run climbs --next --hire, lists the openCI nightly:issues with no linked PR and no worker on them, and hires one worker for each with a fixed prompt: reproduce withbun run builder run apps/<app> bun test <file>, fix, open a PR with "Fixes #N", follow AGENTS.md. It could run from the office queue or a 30-minute timer. It hires nobody for an issue that already has a worker, and it is capped (for example 2 at once) so a broken main can't hire ten. - Guard against a broken main. When the same test fails on 3+ PRs in one day (the report's blocked table), the loop opens a
CI nightly:-style issue for that test right away instead of waiting for the night, since every open PR is red until it is fixed.
The full report follows.
CI report: apps.yml and typecheck.yml, last 7 days
1875 runs (706 apps.yml, 1169 typecheck.yml), 15570 jobs that ran, 2026-11-02 to 2026-11-09; 1 runs left out (GitHub wouldn't list their jobs). Times are America/Los_Angeles. Made by bun scripts/ci-report.ts.
1. Runs: queue, wall time, job-minutes
| workflow | runs | created → first job, p50 / p90 | queue wait per run, p50 / p90 (job-min) | wall, p50 / p90 / max | job-min per run, p50 / total |
|---|---|---|---|---|---|
| apps.yml | 706 | 0.1 / 0.3 min | 0.2 / 44.7 | 1.8 / 7.6 / 43.2 min | 3.0 / 10742 |
| typecheck.yml | 1169 | 0.1 / 0.3 min | 0.0 / 0.1 | 0.8 / 1.1 / 16.8 min | 0.8 / 903 |
Every job: waited 0.4 min for a runner at the median, 3.6 at p90, 28.2 at most; 23064 job-min of waiting against 11645 job-min of work.
Cancelled jobs (a newer push, cancel-in-progress): 910, 952 job-min already spent. Rerun attempts: 100 jobs, 74 job-min.
By hour of day (America/Los_Angeles), all 7 days
Runs with a job waiting for a runner, mean / max; jobs waiting, mean / max; jobs running, max. Today's max in the last column.
| hour | runs waiting | jobs waiting | jobs running, max | today: runs waiting / jobs running, max |
|---|---|---|---|---|
| 00:00 | 0.1 / 3 | 1.7 / 90 | 20 | 1 / 20 |
| 01:00 | 0.5 / 9 | 10.3 / 188 | 20 | 9 / 20 |
| 02:00 | 0.1 / 2 | 2.0 / 58 | 54 | 1 / 20 |
| 03:00 | 0.1 / 2 | 1.0 / 56 | 51 | 0 / 0 |
| 04:00 | 0.0 / 2 | 0.4 / 47 | 13 | 0 / 3 |
| 05:00 | 0.0 / 0 | 0.0 / 0 | 2 | 0 / 0 |
| 06:00 | 0.0 / 1 | 0.0 / 1 | 6 | 0 / 0 |
| 07:00 | 0.0 / 1 | 0.2 / 37 | 20 | 0 / 0 |
| 08:00 | 0.0 / 1 | 0.0 / 1 | 9 | 0 / 0 |
| 09:00 | 0.1 / 17 | 2.1 / 425 | 20 | 17 / 20 |
| 10:00 | 1.1 / 19 | 25.3 / 468 | 33 | 19 / 33 |
| 11:00 | 0.0 / 0 | 0.0 / 0 | 1 | 0 / 1 |
| 12:00 | 0.0 / 1 | 0.0 / 14 | 20 | |
| 13:00 | 0.1 / 3 | 1.7 / 54 | 20 | |
| 14:00 | 0.1 / 3 | 1.1 / 65 | 20 | |
| 15:00 | 0.1 / 3 | 1.1 / 51 | 20 | |
| 16:00 | 0.2 / 5 | 3.4 / 131 | 20 | |
| 17:00 | 0.1 / 2 | 1.0 / 55 | 20 | |
| 18:00 | 0.0 / 2 | 0.2 / 38 | 20 | |
| 19:00 | 0.0 / 2 | 0.3 / 38 | 20 | |
| 20:00 | 0.0 / 2 | 0.4 / 38 | 20 | |
| 21:00 | 0.0 / 1 | 0.1 / 37 | 20 | |
| 22:00 | 0.1 / 3 | 1.0 / 39 | 25 | |
| 23:00 | 0.1 / 3 | 1.6 / 38 | 20 |
While any job was waiting, 19 jobs were running at the median (20 at p90; 54 at the very most, a GitHub burst). That is the account's concurrent-job ceiling: past it a run queues, however fast its tests are. Minutes with a job waiting: 486 of 10019 (5%).
2. Jobs: duration and where it goes
The 30 jobs with the most job-minutes (of 67).
| job | runs | p50 | p90 | job-min | checkout | setup | install | test | other |
|---|---|---|---|---|---|---|---|---|---|
| test skate-tech-park 2/3 | 215 | 4.6 | 5.8 | 942 | 8% | 3% | 1% | 88% | 0% |
| typecheck | 1164 | 0.8 | 0.9 | 903 | 34% | 6% | 29% | 31% | 1% |
| test mythic-masters | 420 | 0.8 | 4.6 | 550 | 16% | 4% | 9% | 70% | 1% |
| containers-plan | 703 | 0.5 | 0.6 | 354 | 45% | 9% | 44% | 0% | 2% |
| test skate-tech-park 1/3 | 215 | 1.5 | 1.8 | 331 | 13% | 4% | 1% | 82% | 0% |
| test alien-disc-golf | 213 | 1.5 | 1.8 | 324 | 13% | 4% | 1% | 81% | 0% |
| test codesplitter 1/3 | 314 | 1.0 | 1.2 | 321 | 53% | 12% | 29% | 4% | 2% |
| test two-fronts 4/4 | 218 | 1.5 | 1.9 | 316 | 34% | 9% | 12% | 43% | 1% |
| test codesplitter 2/3 | 313 | 0.9 | 1.1 | 285 | 55% | 12% | 31% | 0% | 2% |
| test codesplitter 3/3 | 314 | 0.8 | 1.1 | 277 | 56% | 12% | 30% | 0% | 2% |
| test two-fronts 1/4 | 217 | 1.3 | 1.6 | 270 | 14% | 4% | 5% | 76% | 1% |
| test two-fronts 3/4 | 218 | 1.1 | 1.4 | 263 | 18% | 5% | 6% | 70% | 1% |
| test fm-daw | 209 | 1.2 | 1.8 | 257 | 16% | 10% | 9% | 64% | 1% |
| test qcrm | 212 | 1.0 | 1.2 | 245 | 18% | 5% | 13% | 63% | 1% |
| test git-recon | 218 | 1.1 | 1.7 | 232 | 18% | 5% | 1% | 75% | 1% |
| test eqosys | 246 | 0.9 | 1.5 | 232 | 23% | 6% | 14% | 56% | 1% |
| test two-fronts 2/4 | 218 | 1.0 | 1.6 | 227 | 18% | 5% | 6% | 71% | 1% |
| plan | 704 | 0.3 | 0.4 | 223 | 81% | 15% | 0% | 0% | 3% |
| test skate-tech-park 3/3 | 214 | 1.1 | 1.3 | 220 | 19% | 5% | 1% | 74% | 1% |
| test park-hide-seek | 219 | 0.8 | 1.8 | 218 | 20% | 5% | 12% | 62% | 1% |
| test nextjs-example | 202 | 0.9 | 1.1 | 184 | 22% | 6% | 12% | 58% | 1% |
| test splat-desktop | 202 | 0.7 | 0.9 | 176 | 23% | 7% | 2% | 66% | 1% |
| packages | 191 | 0.9 | 1.1 | 163 | 29% | 5% | 25% | 41% | 1% |
| test hippo-tycoon | 212 | 0.7 | 0.9 | 154 | 27% | 17% | 18% | 35% | 2% |
| test astro-example | 202 | 0.6 | 0.8 | 153 | 38% | 9% | 23% | 28% | 2% |
| test pixel-rag | 168 | 0.9 | 1.2 | 151 | 23% | 15% | 14% | 43% | 4% |
| test splat-spray | 219 | 0.6 | 0.9 | 148 | 27% | 8% | 3% | 61% | 1% |
| test motion-midi | 207 | 0.7 | 1.0 | 148 | 30% | 17% | 15% | 35% | 2% |
| test auto-rig | 205 | 0.7 | 0.9 | 146 | 31% | 18% | 16% | 32% | 2% |
| test nuxt-example | 202 | 0.7 | 0.9 | 142 | 27% | 8% | 17% | 47% | 1% |
Across every job: checkout 2906 min (32%), setup 785 min (9%), install 1440 min (16%), test 3919 min (43%), other 129 min (1%).
3. Failures
263 failed jobs. "Untouched" means the PR changed nothing that job tests (its app, a package the app imports, the lockfile): main was broken or the test is flaky. Shards are summed.
| job | failed | of runs | on PRs that touched it | untouched | unknown | distinct PRs hit |
|---|---|---|---|---|---|---|
| test codesplitter | 51 | 314 | 44 | 7 | 0 | 28 |
| typecheck | 20 | 1164 | 0 | 0 | 20 | 18 |
| test qcrm | 20 | 212 | 17 | 3 | 0 | 16 |
| test mythic-masters | 14 | 420 | 12 | 2 | 0 | 12 |
| test two-fronts | 10 | 218 | 2 | 8 | 0 | 3 |
| test alien-disc-golf | 9 | 213 | 7 | 2 | 0 | 6 |
| test eqosys | 8 | 246 | 6 | 2 | 0 | 6 |
| test skate-tech-park | 7 | 215 | 1 | 6 | 0 | 2 |
| packages | 6 | 191 | 3 | 3 | 0 | 3 |
| scripts | 5 | 250 | 3 | 2 | 0 | 4 |
| test agent-office | 5 | 224 | 2 | 3 | 0 | 4 |
| test hippo-tycoon | 5 | 212 | 2 | 3 | 0 | 4 |
| test splat-rooms | 5 | 215 | 3 | 2 | 0 | 4 |
| test park-hide-seek | 5 | 219 | 3 | 2 | 0 | 4 |
| containers-plan | 3 | 703 | 0 | 0 | 3 | 2 |
| test borg | 3 | 202 | 1 | 2 | 0 | 2 |
| test astro-example | 3 | 202 | 1 | 2 | 0 | 2 |
| test elysia-conduit | 3 | 202 | 1 | 2 | 0 | 2 |
| test geoboxq-shopify | 3 | 202 | 1 | 2 | 0 | 2 |
| test react-router-example | 3 | 202 | 1 | 2 | 0 | 2 |
| test motion-midi | 3 | 207 | 1 | 2 | 0 | 2 |
| test elysia-example | 3 | 202 | 1 | 2 | 0 | 2 |
| test auto-rig | 3 | 205 | 1 | 2 | 0 | 2 |
| test fm-daw | 3 | 209 | 1 | 2 | 0 | 2 |
| test artstewengine | 3 | 201 | 1 | 2 | 0 | 2 |
The failing tests by name (from the failed jobs' logs), most often first:
| test | failed | untouched | PRs | days |
|---|---|---|---|---|
| test/demo.test.ts › the demo opens on the shared snapshot, keeps a visitor's edit, and writes only what they touched | 17 | 3 | 14 | 11-05, 11-09 |
| test/round.test.ts › (unnamed) | 7 | 0 | 5 | 11-03, 11-04 |
| test/agent-routes.test.ts › a file dropped into a thread (#3723): everyone sees it, the running agent finds it in its folder with the note, | 7 | 0 | 4 | 11-08 |
| test/agent-routes.test.ts › a run card on the file page hears its run instead of asking (#3737): each log line, then how it ended, and it is | 5 | 0 | 3 | 11-08 |
| test/git-problems.test.ts › the demos list links to the page, and its station animation starts | 4 | 0 | 4 | 11-02, 11-03, 11-08 |
test/layout.test.ts › REFERENCE.md's Layout table matches the files' own Layout lines (run bun run layout) |
4 | 0 | 4 | 11-03, 11-05, 11-08 |
| test/floorguard.test.ts › (unnamed) | 3 | 1 | 3 | 11-08 |
| test/hosts.test.ts › the pinned release still has the request handler and upgrade lines the guard is patched in front of | 3 | 1 | 3 | 11-08 |
| test/power.test.ts › both patch targets are in the pinned release's 3D bundle, once each | 3 | 1 | 3 | 11-08 |
| test/env.test.ts › the pinned release's childEnv, patched, hands workers no secrets | 3 | 1 | 3 | 11-08 |
| test/migrations.test.ts › no new migration shares its number or stamp with another | 3 | 0 | 2 | 11-03 |
| test/tokens.test.ts › GET /api/openapi.json > names every path parameter, and each operation once | 3 | 0 | 3 | 11-03, 11-08 |
| test/heal.test.ts › a required check failing on a branch opens a fix task there; it edits the branch, not main | 2 | 0 | 2 | 11-05, 11-08 |
| test/heal.test.ts › the same failure twice: the next fix task starts from a note saying what was tried, or a person gets it (#1077) | 2 | 0 | 2 | 11-05, 11-08 |
| test/heal.test.ts › a merge guard tripping (git's markers on the branch) opens a fix task, and then it merges | 2 | 0 | 2 | 11-05, 11-08 |
Tests that blocked many PRs at once
The same test failing on 3 or more PRs on one day, most PRs first; "untouched" counts the PRs that changed nothing it tests.
| day | test | PRs blocked | untouched | first → last failure |
|---|---|---|---|---|
| 2026-11-09 | test/demo.test.ts › the demo opens on the shared snapshot, keeps a visitor's edit, and writes only what they touched | 13 | 1 | 01:28 → 02:29 |
| 2026-11-03 | test/round.test.ts › (unnamed) | 4 | 0 | 02:25 → 17:05 |
| 2026-11-08 | test/agent-routes.test.ts › a file dropped into a thread (#3723): everyone sees it, the running agent finds it in its folder with the note, | 4 | 0 | 16:29 → 17:38 |
| 2026-11-08 | test/floorguard.test.ts › (unnamed) | 3 | 1 | 01:02 → 22:21 |
| 2026-11-08 | test/hosts.test.ts › the pinned release still has the request handler and upgrade lines the guard is patched in front of | 3 | 1 | 01:02 → 22:21 |
| 2026-11-08 | test/power.test.ts › both patch targets are in the pinned release's 3D bundle, once each | 3 | 1 | 01:02 → 22:21 |
| 2026-11-08 | test/env.test.ts › the pinned release's childEnv, patched, hands workers no secrets | 3 | 1 | 01:02 → 22:21 |
| 2026-11-08 | test/agent-routes.test.ts › a run card on the file page hears its run instead of asking (#3737): each log line, then how it ended, and it is | 3 | 0 | 16:28 → 17:09 |
4. What to cut, with numbers
Checkout depth. The jobs with fetch-depth: 0 (plan, scripts, containers-plan, containers) check out in 10 s at the median against 8 s for the shallow ones (the test jobs are already shallow). Making them shallow (plan needs only the merge-base: fetch-depth: 1 plus git fetch --depth=200 origin main, or filter: blob:none) saves about 120 job-min (1%).
Per-job overhead. Checkout, setup and install are 44% of all job-minutes: each job pays ~18 s before its first test. 42 app suites finish in under a minute at the median; run them six to a job (one checkout and install for six apps) and they keep five sixths of 2690 overhead job-min: 2242 job-min saved, and 35 fewer jobs in the queue per full run.
Every app on a shared change. 202 of 706 apps runs tested 38+ of 48 apps and spent 8639 job-min on test jobs, 90% of all test job-min. What set them off (202 with a known PR): packages/util 77 (26 importing apps), packages/cli 48 (44 importing apps), bun.lock 40, packages/ui 12 (26 importing apps), scripts/ci-apps.ts 10, packages/rig 9 (2 importing apps), packages/render 7 (1 importing apps), packages/game-controls 6 (10 importing apps). Testing only the apps that import a changed package (and still every app for the lockfile, the workflow and ci-apps.ts) saves 2706 job-min; a nightly all-apps run catches what the import graph misses.
Suites to run on a schedule. A suite qualifies when one run of it costs 1.5+ job-min (shards summed), it ran in 10+ runs, and it caught at most 2 failures on PRs that touched it. Nightly costs one run a day; hourly (like desktop-hourly.yml) 24.
| suite | runs | job-min per run | job-min | caught (touched) | untouched failures | nightly | hourly |
|---|---|---|---|---|---|---|---|
| test skate-tech-park (schedule) | 215 | 6.9 | 1493 | 1 | 6 | 49 | 1167 |
| test two-fronts (schedule) | 218 | 4.9 | 1076 | 2 | 8 | 35 | 829 |
| test codesplitter | 314 | 2.8 | 883 | 44 | 7 | 20 | 472 |
| test mythic-masters | 420 | 1.3 | 550 | 12 | 2 | 9 | 220 |
| test alien-disc-golf | 213 | 1.5 | 324 | 7 | 2 | 11 | 256 |
| test fm-daw | 209 | 1.2 | 257 | 1 | 2 | 9 | 207 |
| test qcrm | 212 | 1.2 | 245 | 17 | 3 | 8 | 194 |
| test git-recon | 218 | 1.1 | 232 | 0 | 2 | 7 | 179 |
| test eqosys | 246 | 0.9 | 232 | 6 | 2 | 7 | 158 |
| test park-hide-seek | 219 | 1.0 | 218 | 3 | 2 | 7 | 167 |
| test nextjs-example | 202 | 0.9 | 184 | 1 | 2 | 6 | 153 |
| test splat-desktop | 202 | 0.9 | 176 | 0 | 2 | 6 | 146 |
| test hippo-tycoon | 212 | 0.7 | 154 | 2 | 3 | 5 | 122 |
| test astro-example | 202 | 0.8 | 153 | 1 | 2 | 5 | 127 |
| test pixel-rag | 168 | 0.9 | 151 | 0 | 2 | 6 | 151 |
Moving the 2 marked suites to nightly saves 2486 job-min (21%); hourly saves 573. The cost: a break in one of them lands on main and shows up on the next scheduled run instead of on its PR, so the scheduled run has to open an issue someone picks up.
Of 11645 job-min in 7 days, these overlap (a scheduled suite was also in the every-app runs), so don't add them up. Fewer and shorter jobs is what shortens the queue: the wait above comes from jobs past the concurrency ceiling, not from slow tests.
Activity
scripts/ci-report.ts: where a week of CI time went (Part of #4136) (#…