Skip to content
This repository was archived by the owner on Nov 4, 2026. It is now read-only.
This repository was archived by the owner on Nov 4, 2026. It is now read-only.

看一下最近的CI为什么都要那么久? #5393

Closed

Description

@os-zhuang

尤其是release

Activity

  1. os-support-ai commented on Aug 20, 2026

    @os-support-ai
    Collaborator

    Measured: the release lane's 43-minute median is 96% one step, then doubled by serialisation

    All figures read from the jobs' own started_at/completed_at, not from an assumed clock. Sample: the last 30 Changeset Release runs (2026-08-19T14:35Z → 2026-08-20T09:58Z), 20 of them successful.

    Where release time actually goes

    successful runs (n=20)
    median 43.3 min
    mean 47.1 min
    min / max 30.1 / 70.9 min

    Step breakdown of run 32350122326 (the 70.9-min one):

    step elapsed
    Set up → Checkout → Corepack → Node → Turbo cache → Install 33 s total (install itself 7 s)
    Run tests 34 min 04 s
    Build packages 35 s
    Create Release PR / Publish to npm 12 s

    Caching is not the problem — pnpm install takes 7 seconds. Effectively the entire job is Run tests.

    Cause 1 — the release job runs the test suite unsharded

    changeset-release.yml calls pnpm test, and root package.json has "test": "vitest run" — the whole suite, one runner.

    ci.yml shards the same suite 4 ways, and its own comment states the trade deliberately:

    That cost shards near-linearly, and the fixed overhead of an extra runner here is only ~30s … so 4 shards cut wall clock ~3x for ~26% more runner minutes.

    Measured both ways on the same commit 9f118b934: sharded on the PR, the whole CI run was 9 min 47 s wall clock (slowest shard 9 min 43 s). Unsharded in release: 34 min. Ratio 3.5x — consistent with the comment's ~3x.

    Cause 2 — releases serialise, so the wait is added on top

    concurrency: ${{ github.workflow }}-${{ github.ref }} with no cancel-in-progress, so the default applies and runs queue rather than cancel.

    The handoff is exact, not inferred:

    • run 32349908848 created 08:39:35, ran 38.0 min → finished 09:17:35
    • run 32350122326 created 08:42:12, but its job was created 09:17:35 and started 09:17:37

    So that run waited 35 min 23 s before starting, then executed 35 min 27 s. The 70.9 min total is half pure queueing. The same pattern repeats at 04:51:35 / 04:51:52 (35.1 min then 69.1 min).

    With roughly 18 merges to main in a working day and ~35 min of execution each, this lane is close to permanently saturated, so each merge inherits the backlog of the ones before it.

    ⚠️ The obvious fix is the wrong one here. ⛔ Do not add cancel-in-progress: true to a workflow whose last step is changeset publish — that cancels a release mid-publish. The lever is to stop spending 34 of the 35 minutes on tests: shard them, or drop the test step from the release job on the grounds that the same commit was already tested on its PR (see the caveat below, which weakens that second option).


    Three findings the measurement turned up alongside — filed here, not acted on

    These are reported, not fixed; two of them are safety issues rather than speed issues.

    1. ⚠️ The merge queue runs no CI at all

    CI runs with event=merge_group: total_count = 0 — with and without a status filter.

    Counter-probed, because a zero is not a reading: the identical call with event=push returns 30 runs, and event=pull_request resolves normally (run 32354059728 reports "event":"pull_request"). The method works; the zero is real.

    Independently corroborated: PR #5334 was enqueued at 09:41:26Z and merged at 09:41:43Z — 17 seconds. No CI run completes in 17 seconds.

    This is the precise hole ci.yml's own comment block was written to close (#3523), including its account of 2026-08-07, when #3503/#3510/#3516 merged with Type Check at conclusion=failure. The comment says the merge_group trigger is subscribed; the run count says it has never produced a build. Whatever the mechanism, the queue is currently a rubber stamp and PR-level checks are the only real gate — which is also why "the commit was already tested on its PR" is a weaker argument than it looks.

    2. ⚠️ CI has not run on push/main since 2026-08-07

    Newest CI run with event=push, branch=main: 2026-08-07T23:55:27Z, ~13 days ago.

    Pushes to main clearly still happen — Changeset Release has 3901 runs, the newest today at 09:58Z. So this is not "pushes stopped". The push lane is where test-coverage lives (if: github.event_name == 'push', unsharded, timeout-minutes: 40), so Codecov has had no report for ~13 days, and the job whose 40-minute budget was sized for the push lane is not running.

    3. release.yml is dead

    The workflow named "Release" (release.yml, id 223155593) has 0 runs. It triggers on push: tags: v*, and nothing pushes those tags. Counter-probed by the same method that returns 3901 runs for Changeset Release.

    Worth knowing when reading this issue's title: "release" in practice means changeset-release.yml, not release.yml.


    ⛔ No changes made — the seat is paused at the maintainer's instruction; this is measurement only. Findings 1 and 2 look like they want their own cards, but I have not filed them pending a word, and filing would need the usual duplicate search first.


    Generated by Claude Code

  2. os-support-ai commented on Aug 20, 2026

    @os-support-ai
    Collaborator

    Correction to finding 2 above, and the optimisation plan it changes

    ⚠️ Finding 2 in my previous comment was wrong

    I wrote that CI had not run on push/main since 2026-08-07. That is false. I took it from a filtered query (branch=main + event=push + status=completed) whose newest result was Aug 7, and I did not counter-probe it against the unfiltered set — the same discipline I applied to the merge_group zero two paragraphs earlier, and skipped here.

    Unfiltered, the most recent 30 CI runs contain 13 push runs on main, the newest today at 09:58:23Z. Codecov is not 13 days stale. I apologise for the noise.

    The correction makes the real picture worse, not better, which is why it matters.

    What actually happens on every merge to main

    lane what runs measured
    PR 4 test shards, no coverage ~9 min 47 s ✅
    merge queue nothing (merge_group runs = 0) 17 s ⚠️
    push → CI Test (coverage) — unsharded, v8 instrumentation; the 4 shards are skipped here by if: github.event_name != 'push' ~39 min
    push → Changeset Release pnpm test — unsharded ~34 min (+ up to ~35 min queued)

    So the full suite runs twice per merge, unsharded, in parallel — roughly 73 minutes of runner time — and neither run gates anything: the CI push run is cancelled by the next merge (cancel-in-progress: true keyed on github.ref), and the release run only delays publishing something already on main.

    Job breakdown of the 09:58:23Z push run (32356587040):

    job elapsed
    Test (coverage) 39 min 26 s — ❌ failure
    Type Check 8 min 57 s
    Build Docs 4 min 53 s
    Build & E2E 59 s
    Changeset Fixed Group Check 6 s
    Test (shard …/4) skipped

    ⚠️ Test (coverage) is 34 seconds away from its own timeout

    That job has timeout-minutes: 40 and just ran 39 min 26 s. Whatever else is decided, this one is urgent on its own: the next natural growth in the suite turns it into a timeout, and a timeout renders as a red that has nothing to do with the commit.

    ⚠️ main is currently red, and the release caused it

    The failing step is a real assertion, not the timeout:

    scripts/__tests__/quick-reference-current-release-4143.test.ts:444
    expected []   received [ "17.5.0" ]
    

    A test pinned to "the current release" went red once changeset version wrote 17.5.0 into the tree. A PR cannot catch this, because the version bump happens after merge. Pinning the expected value instead of reading it from the manifest guarantees one red main per release.


    Proposed optimisation — ⛔ NOT actioned, this is on the manual floor

    Every item below either removes a test step, lowers how often a gate runs, or raises a timeout. That is 门禁削弱, which the standing rules put squarely on the maintainer: "AI 有把 CI 弄绿的结构性动机, 削弱农场必须是人的动作". So this is a proposal for a ruling, not a plan I intend to execute.

    1. Drop pnpm test from changeset-release.yml — biggest win, lowest risk.
    Release goes from ~35 min to ~2 min (install 7 s + build 35 s + publish 12 s), and the queueing disappears with it, because the lane stops being occupied for 35 minutes at a time. What that step tests is a commit already on main — failing it does not keep anything out, it only blocks publishing after the fact. Crucially this is removing the second post-merge full run, not the only one: the CI push lane still runs the whole suite.

    2. Fix Test (coverage). Raise timeout-minutes past 40 immediately regardless of what else is chosen. Then either shard it 4 ways and merge the blob reports (the workflow comment declines this on merge-cost grounds — that trade was priced before the job sat at 39 minutes), or stop running coverage on every push and run it daily / post-release, since Codecov is a trend signal and does not need every merge.

    3. Make the merge queue actually check something — correctness, not speed.
    merge_group: types: [checks_requested] is subscribed in ci.yml, yet merge_group runs = 0. GitHub only dispatches checks_requested when the queue has required checks configured, so the empty set is in the repository ruleset, not the YAML — ⛔ this is a settings change, not a PR. Doing this also strengthens item 1: once the queue validates before merge, a post-merge test run in the release job is even less load-bearing.

    4. Un-pin the release-coupled test so a release stops turning main red — read the current version from the manifest rather than asserting a literal.

    ⚠️ Do not "fix" the release queueing with cancel-in-progress: true: that workflow's last step is changeset publish, and cancelling it mid-publish is worse than waiting.

    Rough saving if 1 + 2 are taken: ~34 min of runner time per merge (~10 h/day at ~18 merges), and release latency from a 43-min median (71 min worst case) down to a couple of minutes.


    Generated by Claude Code

  3. os-support-ai commented on Aug 20, 2026

    @os-support-ai
    Collaborator

    Independent re-measurement is in (#5395) — three of my numbers were wrong, and one conclusion reverses

    The verification card came back. Full evidence is on #5395; this is what changes here, including corrections to my own earlier comments on this issue.

    ⚠️ Correction 1 — my retraction's stated cause was itself unverified

    Above, I retracted the false "CI hasn't run on push/main since Aug 7" finding and blamed a stale filtered query. The re-measurement could not reproduce that: the same filter combination returns the correct newest run through both raw REST and the MCP wrapper, and no single status term reproduces the Aug-7 result.

    So the false finding was real, but the explanation I gave for it is not established. I should have said "I don't know why that query returned Aug 7" instead of supplying a mechanism that sounded right. That is the same failure mode as the "path-based short-circuit" I invented earlier in this session — inventing a cause is not the same as finding one.

    ⚠️ Correction 2 — the release lane does not queue, it drops

    I claimed concurrency without cancel-in-progress serialises releases and that a run can wait ~35 min. The mechanism is confirmed but the magnitude is refuted, and the real behaviour is worse in a different way:

    • median release wait: 5 min 08 s, not ~35 min
    • waiting is 22% of release wall clock, not half
    • 95 of 200 release runs never executed at all — with cancel-in-progress unset, GitHub holds only one pending run per group and discards the rest, rather than queueing them

    That reframes the fix. The concern is not merely latency: release runs are being thrown away, and the one-run-per-group pending slot means merges land without their release run ever starting.

    ⚠️ Correction 3 — merge volume

    I estimated ~18 merges/day. Measured median is 49 pushes to main/day (only 2 of 400 are the changesets bot's own commit). My per-day saving figures are therefore ~2.7× too low.

    Confirmed, and one of them understated

    • Full suite twice per merge, unsharded — confirmed.
    • The 40-minute cap is not a future risk. It has already fired 4 times, with GitHub's own annotation: "The job has exceeded the maximum execution time of 40m0s." I filed this as "34 seconds of margin"; it has been over the line repeatedly.
    • Cancellation — confirmed.
    • Durations confirmed as magnitudes, though each figure sits at a different point of a moving distribution.

    Sharded coverage, priced

    measured
    unsharded 39 min 05 s
    4-way sharded 12 min 04 s (3.24×)
    extra runner minutes +9.2%
    the blob-report merge ci.yml declines on cost grounds 13 seconds

    Equivalence checked rather than assumed: 1287 files both ways, 55432 vs 55395 of 84590 statements, onlyUnsharded=0, onlyMerged=0, delta −0.0437 pp — against a noise floor of 3 statements between two identical sharded runs.

    The stated reason for not sharding costs 13 seconds and buys back 27 minutes.

    ⚠️ The finding that outranks this whole issue — #5402

    Test (coverage) has failed 100% of the time since 2026-08-16T12:48Z: scripts/__tests__/check-package-self-import.test.ts times out at 15000 ms under v8 coverage instrumentation. An ablation isolates coverage, not sharding, as the discriminator — 0 occurrences in 220 PR-lane runs (sharded, no coverage), 50 of 51 completed push-lane failures (unsharded, with coverage).

    Codecov has received nothing for four days, for two independent reasons: the upload step is skipped by its implicit success(), and the live unsharded recipe leaves no coverage/ directory at all when the suite fails.

    ⚠️ This corrects my ACCEPT on #5394. I wrote there that main's red was cleared. It is not: PR #5396 removed the release-drift failure, which was real and recurring, but Test (coverage) was already failing for this independent reason and still is. The push run for that merge (fd227eae1) was itself cancelled, so it never returned a verdict at all — the newest completed verdict on main remains failure. I should have verified a green before saying so.

    Where that leaves the proposal

    The ordering changes. #5402 comes first — the push lane has been reporting nothing but red for four days, so "the CI push lane still runs the whole suite" (my stated safety argument for dropping the release job's tests) has been false the entire time. Sharding is now cheap and well-evidenced. The queueing item should be rewritten around dropped runs rather than latency. All of it still needs a maintainer ruling; ⛔ nothing has been changed.


    Generated by Claude Code

  4. os-zhuang commented on Aug 20, 2026

    @os-zhuang
    ContributorAuthor

    Triage: state → pm:blocked (kept bug). The question is being answered by measurement already in flight:

    Blocked-by: #5395

    #5395 is the dispatched independent re-measurement of the per-merge CI cost model (covers the "为什么都要那么久" half), and the "尤其是 release" half has a measured answer on #5397: the changeset release PR runs no CI at all — every ci.yml run on changeset-release/main is action_required, and the release commit is first validated by the ~39-minute post-merge push build, which is the long red tail being observed. #5397 is now in the decision inbox with the CI-policy options; #10149 (objectstack) carries the same critical-path question for the main repo. This card unblocks and gets its consolidated answer when #5395 reports.


    Generated by Claude Code

  5. os-support-ai commented on Aug 20, 2026

    @os-support-ai
    Collaborator

    Maintainer ruling — 2026-08-20: all recommendations accepted

    Maintainer, 2026-08-20, in chat with the repo:objectui execution seat (session session_01RV6yuVCxymHYE16PL9vQkE). Verbatim: 「接受你的所有建议,现在开始处理。」

    This clears the 门禁削弱 manual floor for the specific items below and nothing else. Each item's scope is what is written here; anything discovered mid-implementation that goes beyond it returns to the maintainer under the roll-back clause rather than being decided by a seat.

    Ruled, in execution order

    The order is load-bearing, not a preference. Items 2 and 3 remove or replace validation, and the safety argument for each depends on the item before it actually working.

    # item state
    1 Fix Test (coverage) — check-package-self-import.test.ts times out at 15000 ms under v8 coverage; failing 100% since 2026-08-16, Codecov dark 4 days #5402, first, blocks the rest
    2 Shard Test (coverage) 4 ways (39m05s → 12m04s, +9.2% runner, 13 s blob merge) and fix the upload's implicit success() — both live in the same test-coverage job new card, Blocked-by: #5402
    3 Remove pnpm test from changeset-release.yml, and stop concurrency dropping runs (95 of 200 never executed) — same file, one card. ⛔ Not cancel-in-progress: true: the last step is changeset publish new card, Blocked-by: item 2
    4 Delete release.yml — 0 runs, ever; tag-triggered on v* which nothing pushes new card, independent
    5 Fix assertCanonicalVitestInvocation — does not cover the 17 packages carrying their own vitest.config.ts, contrary to its own docstring new card, independent

    ⚠️ Why 3 comes last, and why my original ordering was wrong. I first proposed removing the release job's tests as the top item, arguing the CI push lane still ran the whole suite as a backstop. Item 1 proves that backstop has been red 100% of the time since 2026-08-16. Removing release-side tests before 1 and 2 land would leave main with no working post-merge validation at all.

    ⛔ Not actionable by any seat — still the maintainer's, in repository settings

    The merge queue runs no CI (#4986, type:Bug, on pm:on-hold since 2026-08-17; re-verified today: merge_group runs = 0, and PR #5334 went enqueued → merged in 17 seconds). ci.yml already subscribes merge_group: types: [checks_requested]; GitHub only dispatches that event when the queue has required checks configured, so the empty set is in the repository ruleset. ⛔ No PR can close #4986. Until it is configured, the PR lane is the only validation any change to main receives — which is also the standing premise AGENTS.md §9 relies on and which has never been true.

    The related #5397 (the release PR itself runs no CI, because a GITHUB_TOKEN-authored PR starts no workflows) is filed and unassigned; it is not in the ruled set above.

    Also outstanding, unrelated to the ruling

    The verification card's throwaway branch claude/issue-5395-ci-cost-model-verification could not be deleted — this session's proxy returns 403 for both git push --delete and the REST ref delete. It was emptied instead (a07624e22, empty diff against main); someone with delete rights should remove it.


    Generated by Claude Code

  6. added
    domain:devxobjectui devx stream: fix lands on .github/, scripts/ or release pipeline — devx lane cross-repo
    on Aug 23, 2026
  7. os-zhuang commented on Aug 23, 2026

    @os-zhuang
    ContributorAuthor

    Triage: domain:devx (lane only — pm:blocked untouched). CI wall-clock is an engineering-surface question: whatever the answer turns out to be, it lands in workflow configuration, caching, or shard layout.

    ⚠️ Two things a claiming seat should know, because this card is thin:

    1. The body names no file and no measurement — it is a question, not a scoped card. The first deliverable is a measurement (where the time actually goes), and it is legitimate to come back with that and a re-scoped card rather than a fix.
    2. Several already-routed domain:devx cards are plausible partial answers and should be read first rather than rediscovered: finding: moduleSpecifiers() passes setParentNodes: true and nothing reads a parent — ~25% of two gates' dominant cost, byte-identical output without it #5410 (a gate passing setParentNodes: true for nothing — ~25% of two gates' dominant cost), ci.yml's concurrency cancels the push-lane coverage job mid-run, so Codecov gets nothing on a merge that is followed by another merge #5422 (concurrency cancelling the coverage job mid-run), Test (coverage) is red on every push to main because CODECOV_TOKEN is unset — the four shards are green, the upload is rejected #5436 (coverage upload rejected), 合并队列声称「已强制」却从未产生过一次 merge_group 构建(repo-wide 0),必需集实测不含 4 个 shard / Type Check / Lint —— #3523 的第 3 步从未落地,而 AGENTS.md §9 已按「队列会替你兜住」反转了 auto-merge 禁令 #4986 (the merge queue that has never produced a build). If the measurement lands on one of those, close this as a duplicate rather than opening a parallel fix.

    Routed via the maintainer direct-dispatch channel, this session, verbatim: 「然后 批 4–5」. PM session session_0124Qg8rLvpXnQDwCmpKUmaJ. objectui three-stream split (maintainer 2026-08-21); not a Routine triage fire — the triage seat may re-grade.


    Generated by Claude Code

  8. claude commented on Aug 23, 2026

    @claude
    Contributor

    Re-grade (triage seat, session session_01LsWeHbPzR3i6mmonFfGykk, 2026-08-23 ~14:2xZ): Blocked-by: #5395 resolved — the independent re-measurement was delivered and ACCEPTed (its report is the artifact; it corrected the cost model, e.g. release-lane median wait is 5m08s / 22% of wall clock, not ~35min). What remains on this card is not measurement but a ruling: the priced options cluster (2026-08-20T14:25Z comment) includes gate-shape changes — removing pnpm test from changeset-release.yml and altering concurrency-drop behavior are gate-weakening-class calls, which sit on the manual floor. pm:blocked → needs-user-decision in the same stroke. Note for the decision: option pieces already landed independently (coverage was 4-way sharded; CODECOV_TOKEN half resolved via #5436) — re-read the 14:25Z cluster against #5395's corrected numbers before ruling; the un-ruled residue is essentially option 3.

    四维分析(补充,中文;选项与实测价格见卡上 2026-08-20T14:25Z 评论,此处不复述):

    • 实际业务需求 — CI 时长直接吃每次合并的落地延迟;测量已证瓶颈与等待结构,修与不修都有明码标价。
    • 项目长远合理性 — 分片与并发治理是可持续形态;从 release 工作流里删测试是用安全换速度,需要显式裁决而非顺手改。
    • 防 AI 写代码犯错 — 门禁削弱类改动必须人裁:AI 有把 CI 弄绿的结构性动机,这正是人工地板存在的理由。
    • 创业阶段不扩散需求 — 已落地的分片解决了大头;剩余选项只在维护者仍觉得慢时才值得做,不默认扩散。

    推荐:若当前 CI 时长已可接受,关闭本卡(测量归档在 #5395);若仍要提速,先做并发丢跑治理(不削门禁),changeset-release.yml 删测试一项单独裁。


    Generated by Claude Code

  9. claude commented on Aug 23, 2026

    @claude
    Contributor

    Maintainer ruling recorded (atomic four-part write)

    Ruled 2026-08-23, live PM chat, verbatim: 「10950 不考虑存量,其他接受你的建议」 — this card falls under the accepted recommendations. Ruled: current CI duration is acceptable — closing. The question is answered by the #5395 measurement (release-lane median wait 5m08s / 22% of wall, not ~35min; coverage now 4-way sharded; CODECOV half resolved via #5436). The un-ruled residue — removing pnpm test from changeset-release.yml (gate-weakening) — is NOT approved. If CI feels slow again, reopening is free: the first step then is concurrency-drop hygiene (95/200 runs never executed), which does not weaken any gate.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdomain:devxobjectui devx stream: fix lands on .github/, scripts/ or release pipeline — devx lane cross-repo

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions