[measurement] Independently re-verify the per-merge CI cost model before any gate change is ruled on — and price the sharded-coverage option

Filed by the repo:objectui execution seat as the confirmation half of #5393. This card lands no gate change. Its deliverable is evidence, posted on this thread, that the maintainer can rule on.

Why this exists

#5393 proposes removing a test step from the release workflow and changing how Test (coverage) runs. Both are 门禁削弱 and sit on the maintainer's floor. Before anyone rules, the numbers behind the proposal need to survive an independent re-measurement — the PM's own investigation already produced one wrong finding (it claimed CI had not run on push/main since 2026-08-07; that came from a filtered API query that was never counter-probed against the unfiltered set, and it is false). Treat every figure below as a claim to check, not a premise.

Claims to confirm or refute

Each one: report the measurement, the method, and whether it confirms or refutes. A refutation is a good outcome, not a failed run.

"The full suite runs twice per merge, unsharded." On push/main, ci.yml's Test (coverage) runs unsharded (the 4 shards are skipped by if: github.event_name != 'push') and changeset-release.yml runs pnpm test unsharded. Confirm both, and confirm they overlap in time rather than one gating the other.
The durations. Claimed: Test (coverage) ≈ 39 min; release Run tests ≈ 34 min; PR lane (4 shards) ≈ 9 min 47 s. Sample enough runs to give a median and a spread — ⚠️ one run is an anecdote. Compute elapsed from each job's own started_at/completed_at, ⛔ never from an assumed clock.
The timeout margin — this sets the urgency. Test (coverage) has timeout-minutes: 40 and one observed run took 39 min 26 s. Is that typical or an outlier? Report the distribution over recent runs and how many came within, say, 3 minutes of the cap. If it is typical, this is already a live flaky-red source and should be said plainly.
The release queueing. Claimed: concurrency without cancel-in-progress serialises releases, so a run can wait ~35 min before its job starts. Verify by comparing each run's created_at against its job's created_at (the run-level run_started_at does not expose this — it equals created_at even when the job waits). Report how much of release wall-clock is waiting vs executing.
Cancellation. Claimed: most push/main CI runs end cancelled because cancel-in-progress: true is keyed on github.ref. Confirm, and report what fraction of push-lane runs actually complete — a lane that is usually cancelled is not providing the coverage its existence implies.

Price the sharded-coverage option

ci.yml's comment declines to shard the coverage job on the grounds that it would require a blob-report merge and "nothing blocks on this job". That trade was priced when the job was shorter. Re-price it:

What does a 4-way sharded coverage run actually cost in wall clock, and does the merged report stay equivalent for Codecov (same totals, no lost files)?
Prototype it on a branch and measure. ⛔ Do not merge it, and do not open a PR that could be picked up as ready-to-land — this card produces a number, not a landing.

Rules for this card

⛔ No gate weakening, and no gate changes merged. Do not delete a test step, lower a threshold, raise a timeout, skip or quarantine a test, or change what is required. All of that is the maintainer's call on 看一下最近的CI为什么都要那么久？ #5393. If you become convinced one is right, say so with evidence — do not do it.
⛔ Do not "fix" main here. It is red for a separate reason, owned by main is red: QUICK_REFERENCE.md "Current Release" still says 17.5.0 after the 17.6.0 release bump — and this is the third instance of one recurring mechanism #5394. Serial note: that card touches QUICK_REFERENCE.md and possibly the release tooling; this one must not.
A zero is not a reading. Any total_count: 0 must be counter-probed with a term/filter combination known to return results through the same method, and the counter-probe reported. The wrong finding named above is precisely what skipping this produces.
Filtered list queries in this repo's tooling have returned stale or wrong newest-first results; cross-check anything load-bearing against an unfiltered read.
Report elapsed from the jobs' own timestamps.

Deliverable

One comment on this card: a table per claim (confirmed / refuted, measured value, method), the sharded-coverage price, and anything the investigation turned up that neither #5393 nor this card anticipated. No merge, no PR to land.

Related: #5393 (the investigation and the proposal), #4986 (the merge-queue half — already filed, on hold, ⛔ not in scope here), #5394 (main red, separate owner).

Claim: PM loop round 20
Session: session_01RV6yuVCxymHYE16PL9vQkE
Branch: claude/issue-5395-ci-cost-model-verification
Worktree: objectui-issue-5395
Domain: repo:objectui
File surface: none in main — this card produces measurements, not a landing. A throwaway branch may carry a sharded-coverage prototype of .github/workflows/ci.yml for timing only; ⛔ it is not to be merged and no PR is to be opened for it (stop on breach; explain in the report)
Container & model: M, mode:subagent, model: opus
Clause-②: no — no contract, no public surface, no merged change at all
Serial constraints cleared: disjoint from #5394 by construction (that card owns QUICK_REFERENCE.md and the release tooling; this one is forbidden from touching either, and from "fixing" main). #4986 owns the merge-queue/ruleset half and is explicitly out of scope here. No open PR touches .github/workflows/ci.yml.

Generated by Claude Code

Independent re-measurement of the #5393 cost model

Re-measured from scratch against the raw REST API (api.github.com, curl + token), not
through a wrapper, with every load-bearing figure taken from unfiltered reads and every
elapsed computed from the jobs' own started_at / completed_at. Current time was read
from the API Date response header rather than assumed — a mistake I made once mid-run and
caught, which is exactly the failure mode this card names.

Samples (all unfiltered GET /actions/workflows/{file}/runs, then classified locally):

set	n	window
`ci.yml` runs, all events	1200	2026-08-12T00:05:51Z .. 2026-08-20T09:58:23Z
of those, `push`/`main`	378	same
jobs fetched for those	2238	same
`ci.yml` `pull_request` runs sampled	220 (133 with 4 real shards)	2026-08-18 .. 2026-08-20
`changeset-release.yml` runs	400 (200 with jobs fetched)	2026-08-12T04:23Z .. 2026-08-20T09:58Z

Verdict table

#	claim	verdict	measured
1	full suite runs twice per merge, unsharded, overlapping	confirmed	shards `skipped` in 121/121 push runs; `Test (coverage)` present in 121/121; 89/89 commit pairs overlap in wall clock, median overlap 11m09s
2	39 min / 34 min / 9m47s	confirmed as magnitudes, corrected as statistics	coverage 37m32s median (3-day, n=35); release `Run tests` 31m14s median (n=104); PR slowest shard 9m17s median (n=133)
3	`Test (coverage)` is 34 s from its 40-min cap	confirmed, and stronger than claimed	the cap has already fired 4 times; 21/35 recent runs land within 3 min of it
4	`concurrency` without `cancel-in-progress` makes runs wait ~35 min	mechanism confirmed, magnitude refuted	median wait 5m08s, not ~35 min; 22% of release wall clock is waiting, not half; and 95/200 runs never ran at all
5	most `push`/`main` CI runs end `cancelled`	confirmed	271/378 = 71.7% cancelled; only 28.3% reach a verdict

Everything below shows the method for each.

Claim 1 — the suite really does run twice per merge, in parallel — CONFIRMED

Static side, from main:

ci.yml test (the 4 shards) carries if: github.event_name != 'push'.
ci.yml test-coverage carries if: github.event_name == 'push' and timeout-minutes: 40.
changeset-release.yml runs pnpm test; root package.json has "test": "vitest run" — unsharded.

Measured side, over the 124 push/main runs in the recent window (121 of which produced jobs;
3 were cancelled before any job existed):

	result
runs where the shard matrix is present but `skipped`	121/121
runs containing a `Test (coverage)` job	121/121
`pull_request` runs where `Test (coverage)` is present but `skipped`	153/153

So the two lanes are exactly complementary, as the YAML says.

Do they overlap, or does one gate the other? Pairing the two lanes by head_sha gives 89
commits where both lanes produced a real test execution. 89/89 overlap in wall clock —
median overlap 11m09s, max 37m41s, and not one pair where the release job started only after
the coverage job had finished. 86/89 pairs were even created at the identical push timestamp.
Neither gates the other; they are two independent full runs of the same suite on the same commit.

Most recent pair, run 32356587040 and 32356587006, sha 59f61cfb8:

CI  Test (coverage): 2026-08-20T09:58:43Z -> 2026-08-20T10:38:09Z   39m26s  failure
Rel pnpm test      : 2026-08-20T10:17:46Z -> 2026-08-20T10:51:01Z   33m15s  failure
overlap: 20m23s

Claim 2 — the durations — CONFIRMED as magnitudes, CORRECTED as statistics

Each figure in #5393 is a real observation, but each sits at a different point of its
distribution, and one of the three distributions is moving.

lane	claimed	measured median	spread	n
`ci.yml` `Test (coverage)` job	~39 min	37m32s (last 3 days)	min 22m13s, p25 35m43s, p75 38m46s, max 39m47s	35
`ci.yml` `Test (coverage)` job		32m42s (last 8 days)	p10 27m27s, p90 38m21s	109
release `Run tests` step	~34 min	31m14s	job total 33m18s median	104
PR lane, slowest of 4 shards	9m47s	9m17s	p10 8m51s, p90 9m39s, p95 9m45s, max 10m30s	133

39 min for the coverage job sits between p75 and p90 of the current window — high, but not
an outlier, and see claim 3.
9m47s for the PR lane is the p95, not the median. The median is 9m17s. Both numbers are
right; only the word "typical" would be wrong.
Cancelled jobs are excluded from every duration above. A cancelled job's elapsed is a
truncation, not a duration. Including all 121 recent coverage jobs rather than the 35 that
finished puts the median at 16m44s instead of 37m32s — less than half — so this choice is
load-bearing, not cosmetic.

Per-step medians, which settle a side question from #5393:

step	`Test (coverage)`	PR shard	release job
the test step itself	37m10s	8m20s	31m14s
everything else combined	0m25s	0m29s	1m33s
`Install dependencies` alone	0m06s	0m06s	0m06s

pnpm install is 6 seconds. #5393's "caching is not the problem" is confirmed with a larger
sample: fixed overhead on a runner in this repo is 25-29 s total.

That also prices sharding on its own, without coverage: 4 shards at 8m20s = 33m20s of runner
time versus 31m14s unsharded — sharding the plain suite costs about 7% more runner minutes
and returns 3.7x in wall clock.

Claim 3 — the timeout margin — CONFIRMED, and the situation is past the point the claim describes

#5393 says the job is "34 seconds away from its own timeout". That understates it.
The 40-minute cap has already fired four times, and GitHub says so in its own words.

run	job elapsed	GitHub annotation
32350122311	40m11s	`The job has exceeded the maximum execution time of 40m0s`
32333429757	40m16s	same
32328584507	40m04s	same
32280945493	40m06s	same

These surface in the API as conclusion: cancelled, which is why a scan for failure misses
them. They are not cancel-in-progress kills, and that was checked rather than assumed:
cancel-in-progress can only fire when a newer push/main run exists, and for each of the
four the next newer run was created after the job had already ended (by 19m00s, 187m18s,
38m57s and 0m11s respectively). A genuine cancel-in-progress kill looks different — e.g. run
32355187526's coverage job ran 16m53s and ended 16 s after the newer run was created.

Distribution against the cap, over the 35 coverage jobs that ran to completion in the last 3 days:

	count
within 3 min of the cap (37m00s or longer)	21/35 (60%)
within 5 min of the cap (35m00s or longer)	27/35 (77%)
past the cap (timeout kills, counted separately)	4

So: typical, not outlier. And the reason is visible in the trend — the job's duration is
growing about a minute and a quarter per day:

day	median completed duration	max	n
2026-08-12	28m44s	30m07s	16
2026-08-13	30m12s	32m57s	17
2026-08-14	32m17s	33m37s	7
2026-08-15	32m42s	34m40s	14
2026-08-16	34m07s	35m38s	13
2026-08-17	35m56s	37m48s	14
2026-08-18	36m57s	39m17s	13
2026-08-19	38m24s	39m14s	12
2026-08-20	39m32s	39m47s	3

The PR lane over the same days moves far less (9m12s, 9m21s, 9m40s), so this is mostly the
coverage lane's own growth, not a general runner slowdown. Inference, labelled as such: at
this slope the median crosses 40 min within about a day, and the four timeouts above are the
leading edge of that, not bad luck.

Claim 4 — release queueing — MECHANISM CONFIRMED, MAGNITUDE REFUTED

First, the card's methodological warning is confirmed exactly: across all 105 executed release
runs, run_started_at minus created_at is 0 seconds in 105/105 cases, including runs
that waited 35 minutes. The run-level field does not expose the wait; the job's created_at
does.

Measured with the job's own created_at, over the 104 release jobs that ran to completion:

	median	mean	min	max	p90
queue wait (`run.created_at` to `job.created_at`)	5m08s	9m15s	0m00s	35m50s	26m08s
executing (`job.started_at` to `job.completed_at`)	33m18s	32m39s	19m54s	37m56s	35m37s
total	38m02s	41m57s	19m57s	70m52s	57m25s

Aggregate: 962 minutes waiting against 3395 minutes executing — 22% of release wall clock is
queueing, not half. 72/104 runs waited more than 60 s.

#5393's specific example reproduces exactly (run 32350122326: waited 35m23s, executed 35m27s,
70.9 min total) — but that is the maximum of the sample, not the typical case. The sentence
"the 70.9 min total is half pure queueing" is true of that run and false of the lane.

And the model behind the claim is wrong in a way that matters. #5393 says "each merge
inherits the backlog of the ones before it". It does not. concurrency without
cancel-in-progress keeps at most one run pending per group; a newer arrival cancels the
pending one, never the running one. So the lane is a one-slot server with a one-slot waiting
room, and merges arriving when both are occupied are dropped, not queued.

Two independent predictions, both hold at 100%:

95 of 200 release runs have no job record at all — they never executed anything. Every
one is cancelled, and 95/95 were cancelled within 90 s of the next run's creation
(typically 1-2 s). Example: run 32326250013 created 02:53:28Z, cancelled 02:56:45Z; the next
run was created 02:56:44Z.
0 of 104 adjacent executed release jobs overlap in time.

Consequence for the ruling: the release lane does not have an unbounded backlog to drain, so
"stop spending 34 of the 35 minutes on tests" would cut the 33m18s execution and most of the
5m08s median wait, but it would not be recovering a 35-minute queue on every merge.

⚠️ #5393's warning against cancel-in-progress: true on this workflow stands and is
reinforced by the above: the runs currently being cancelled are the pending ones, which have
published nothing. cancel-in-progress: true would start cancelling the running one, whose
last step is changeset publish.

Claim 5 — cancellation — CONFIRMED

Unfiltered, 378 push/main ci.yml runs over 8 days:

conclusion	n	share
cancelled	271	71.7%
success	56	14.8%
failure	51	13.5%

Only 28.3% of push-lane runs reach a verdict at all. At job level for Test (coverage)
specifically: 264 cancelled / 58 success / 51 failure — a verdict in 29.2% of runs.

Narrowed to the last 3 days (n=124): 73.4% cancelled, 33 failures, and zero successes.
A lane that is cancelled 7 times out of 10 is not providing the coverage its existence implies —
confirmed. The next section shows it is worse than that.

Findings neither #5393 nor this card anticipated

A. ⚠️ `Test (coverage)` has not passed since 2026-08-16, and Codecov has had nothing since

This is the finding I would put in front of the maintainer first, because it changes what the
job is worth.


last successful `Test (coverage)` job	2026-08-16T10:56:13Z, run 31942979071
last successful `Upload coverage to Codecov` step	the same run — 58/373 coverage jobs ever reached it in the window, none after that one
completed coverage jobs since	51, all failures, 0 successes

The Upload coverage to Codecov step has no if:, so it carries the implicit success() and
is skipped whenever the test step fails — it is skipped in 311 of 373 coverage jobs and in
120 of the 121 most recent ones. So the entire justification in ci.yml's comment —
"one unsharded run so Codecov still receives a single complete report" — has been delivering
nothing for four days.

And it is one test. Reading GitHub's check-run annotations for every one of those jobs:

failing test file	occurrences among the 51 completed failures
`scripts/__tests__/check-package-self-import.test.ts`	50
`scripts/__tests__/quick-reference-current-release-4143.test.ts`	3
three others (one run, 2026-08-17)	1 each

The message is always the same:

[unit] scripts/__tests__/check-package-self-import.test.ts
  > a package name that is not a module edge is not a finding
  > the two live specimens in this repository stay unflagged
Error: Test timed out in 15000ms.
  at scripts/__tests__/check-package-self-import.test.ts:140:3

First occurrence 2026-08-16T12:48:02Z — one run after the last green one at 10:56Z. Present in
every completed failure since.

It does not fail on the PR lane. Same method, same annotation API, over the failing PR shard
jobs in the same window: check-package-self-import.test.ts appears 0 times. Counter-probe
for that zero, because a zero is not a reading: the identical scan over the identical jobs does
return annotations — 7 of them, naming vite-alias-closure.test.ts,
AppContent.systemHubRoutes.test.tsx, vite-objectstack-spec-dist.test.ts,
check-type-check-coverage.test.ts and check-changeset-no-major.test.ts — and the same call
against the push lane returns 50 hits for the file in question. The method works; the zero is real.

Inference, labelled as such: this is a 15-second per-test budget being crossed under v8
coverage instrumentation — the exact shape AGENTS.md describes under
"flaky 测试:先找竞态,别调超时", where an unbounded cost lands inside a bounded window. It is
not a main-content bug and it is not #5394: the QUICK_REFERENCE.md drift appears in 3 of
373 coverage jobs, all recent; this one has been there for four days.

Sharding would not fix this — measured by ablation. The prototype run gives the third cell
of the table, so the discriminator can be identified rather than guessed:

lane	sharded?	v8 coverage?	does `check-package-self-import` time out?
PR lane, production, 220 runs sampled	yes, 4 ways	no	no — 0 occurrences
push lane, production, 373 coverage jobs	no	yes	yes — 50 of 51 completed failures
prototype run 32364577842, same commit	yes, 4 ways	yes	yes — present in the merged report, and shards 3 and 4 exited 1

Holding coverage on and varying sharding changes nothing; holding sharding on and varying
coverage flips it. The v8 instrumentation is what pushes that test past its 15-second budget,
not the size of the run. So sharding the coverage job buys the wall clock and the timeout
margin, but it does not make the lane green — that test needs its own fix either way.

Two consequences for the ruling on #5393:

Proposal 2's "either shard it or run it daily" is not just a cost decision — whichever is
chosen has to make this test pass, or Codecov stays dark either way.
The argument for proposal 1 ("the CI push lane still runs the whole suite") is weaker than it
reads: the push lane currently reaches a verdict in 29% of runs and that verdict has been
failure every time for four days. The release job's pnpm test is, right now, the only
post-merge full-suite run that ever comes back green.

B. The merge cadence in #5393 is roughly 2.7x low

#5393 assumes "roughly 18 merges to main in a working day". Every push to main creates
exactly one Changeset Release run (that workflow has no paths-ignore), so counting those
runs counts pushes:

day	pushes to `main`
2026-08-13	57
2026-08-14	13
2026-08-15	41
2026-08-16	49
2026-08-17	74
2026-08-18	76
2026-08-19	36

Median 49/day. And only 2 of the 400 sampled pushes are the changesets bot's own
chore: release packages commit, so this is real merge traffic, not the release lane feeding
itself. Any per-day saving estimate in #5393 should be scaled accordingly.

Billed runner time actually consumed by the two post-merge suite runs, summing real job
durations rather than modelling: 96.2 runner-hours over ~2.7 days, about 36 runner-hours per
day — and that is with 71% of coverage jobs being cancelled early. Nominal cost if they all
ran to completion is 68m24s of test-step time per merge.

C. #5393's account of its own false finding is not reproducible

The card asks me not to inherit the PM's mistake, so I tried to reproduce it. The retraction in
#5393 attributes the wrong Aug-7 figure to the filter combination branch=main + event=push +
status=completed returning a stale newest result. That query is not stale. Run against raw
REST just now:

branch=main&event=push&status=completed  -> total_count=3847, newest 2026-08-20T09:58:23Z

correctly ordered, newest first. The same filters through the MCP Actions wrapper return the same
newest run. I probed each status value one at a time (success, failure, cancelled,
in_progress, queued, waiting, action_required, neutral, timed_out, skipped,
stale) and none yields 2026-08-07 as the newest result.

So the finding was indeed false, but the stated cause is unverified. What is verified is the
remedy the card prescribes: the unfiltered read cross-checks correctly every time, and it is what
caught this. Worth recording so the next agent does not "avoid the stale filter" and think itself
safe.

One incidental oddity found while probing, reported because it is the same class of trap:
status=queued returns total_count: 1 with an empty result page. A count is not a reading
either.

D. `Test (coverage)` failing does not turn the check red for anything

Not a new claim, just confirmation of the premise both other cards rest on: 89/89 pairs show the
two lanes running concurrently and independently, ci.yml's push lane is not judged by branch
protection or the merge queue (its own comment says so), and 271/378 push runs are cancelled
mid-flight with no consequence. Nothing blocks on it — which is what makes four days of red go
unnoticed.

Pricing the sharded-coverage option — measured, not modelled

ci.yml's comment declines sharding the coverage job because it "isn't worth the blob report
merge". I built the sharded version on a throwaway branch and ran it against the unsharded
production recipe on the same commit, in the same run, on the same runner class, so the
comparison has no day-to-day drift in it.

Branch claude/issue-5395-ci-cost-model-verification, commit 6a4b435e4 (a checkout of
main at 59f61cfb8 plus the throwaway instrument), run
32364577842:

job	test step	job wall clock
unsharded coverage — the production recipe verbatim	38m40s	39m05s
sharded coverage 1/4	10m16s	10m46s
sharded coverage 2/4	10m45s	11m17s (slowest)
sharded coverage 3/4	10m05s	10m34s
sharded coverage 4/4	8m47s	9m16s
merge the 4 blob reports	0m13s	0m47s

The price:

	unsharded (today)	4-way sharded	delta
wall clock (critical path)	39m05s	12m04s	3.24x faster, 27m01s saved per merge
runner minutes	39m05s	42m40s	+9.2%
the "blob report merge"	n/a	13 s of compute, 47 s of job	—

Against the 3-day production median of 37m32s rather than this one baseline, it is still 3.11x
and 25m28s saved.

Reproduced. A second run on the same commit
(32368063518) measured the
four shards at 10m46s / 11m13s / 9m23s / 8m31s — slowest 11m13s against run 1's 11m17s — and its
unsharded arms at 39m25s and 40m13s. Two independent samples, same answer.

So the cost the comment declines to pay is 13 seconds. The merge is
vitest run --merge-reports --coverage; the shards are the existing command plus
--reporter=blob --shard=N/4. The ~26% more runner minutes the comment predicts for sharding
measures 9.2% here, because coverage makes the per-shard work larger relative to the fixed
25-second runner overhead.

Sharding also moves the job to a comfortable distance from any timeout — the slowest shard is
11m17s against timeout-minutes: 20 on the existing shard job.

Does the merged report stay equivalent for Codecov? — yes, no dropped files

Run 1 could not answer this: the unsharded arm produced no coverage file at all. So run 2
(32368063518) puts both
arms through the identical blob-plus-merge path, so the only difference between the two
reports is one process versus four.

	unsharded, 1 process	4 shards merged	delta
files in the report	1287	1287	0 — `onlyUnsharded=0`, `onlyMerged=0`
statements	55432/84590 = 65.5302%	55395/84590 = 65.4865%	-37 hits, -0.0437 pp
functions	11593/18972 = 61.1058%	11589/18972 = 61.0848%	-4 hits, -0.0211 pp
branches	47720/80165 = 59.5272%	47695/80165 = 59.4960%	-25 hits, -0.0312 pp
files with fewer covered statements after merging	—	3	`DirectoryPage.tsx` 6 to 3, `ResourceEditPage.tsx` 464 to 431, `AuthProvider.tsx` 193 to 192

No file is dropped, and the totals move by four hundredths of a percentage point against
thresholds set at 40/33/30/40. For a trend signal that is immaterial.

Is the 37-statement gap sharding, or noise? I have a noise floor, because the two prototype runs
each produced a merged 4-shard report from the same commit: statements 55398 vs 55395, functions
11591 vs 11589, branches 47694 vs 47695 — so an identical configuration wanders by about 3 hits
between runs. The unsharded-versus-sharded gap is roughly ten times that, so a small systematic
difference is likely real, concentrated in three files (one of them, ResourceEditPage.tsx,
accounts for 33 of the 37). Inference, labelled as such: those look like files whose covering
tests land in different shards and no longer share a module instance. It does not lose files and
it does not move the reported percentage meaningfully, but it is not bit-identical and should not
be described as such.

⚠️ Reproduced: the live unsharded recipe writes no coverage report at all when the suite fails

Both prototype runs, on the same commit, with the production command
pnpm test:coverage --coverage.reporter=json --coverage.reporter=text:

run	job	outcome
32364577842	unsharded live, 39m05s	`NO COVERAGE FILE at coverage/coverage-final.json`
32368063518	unsharded live, 39m25s	`NO coverage/ DIRECTORY AT ALL after the live unsharded run`

Two for two. Meanwhile the blob-plus-merge path produced a full report from the same tree with
the same failing tests, every time.

So there are two independent reasons Codecov has received nothing since 2026-08-16, not one:
the Upload coverage to Codecov step is skipped by its implicit success(), and on this
evidence there would have been no file for it to upload anyway. As long as the suite is red, the
current job cannot produce a coverage report at all.

Incidental, from the same run

PROTO unsharded via blob took 40m13s and the live arm 39m25s — on a checkout of main, on
an ordinary runner, both at or past the production job's timeout-minutes: 40. That is claim 3
reproducing outside the production lane.

The blob reports themselves are small: 4 artifacts of 5.86-5.99 MiB, uploaded in 3 s each and
downloaded in 4 s; the merged `coverage-final.json` is 3.83 MiB. Adding those to the 13 s merge
step, the total price of "the blob report merge" that `ci.yml` declines is under a minute.

What I could not measure

Actions job logs and artifacts. Both redirect to productionresultssa12.blob.core.windows.net,
which this session's egress proxy refuses with CONNECT tunnel failed, response 403. Everything
above therefore comes from the REST job/step/annotation surfaces, and the prototype was built to
emit its results as annotations for the same reason. Per-test timings inside the suite, and
vitest's own Duration (transform / setup / collect / tests / environment) breakdown, are not
available to me — 看一下最近的CI为什么都要那么久？ #5393's figures for those are unverified here, neither confirmed nor refuted.
Why the live unsharded run writes no coverage report on a failing suite. That it happens
is measured and reproduced (2 of 2 runs); the mechanism is not, and identifying it would need
the job logs above.
Anything about the merge queue or the repository ruleset. That is 合并队列声称「已强制」却从未产生过一次 merge_group 构建(repo-wide 0),必需集实测不含 4 个 shard / Type Check / Lint —— #3523 的第 3 步从未落地,而 AGENTS.md §9 已按「队列会替你兜住」反转了 auto-merge 禁令 #4986 and I stayed out of
it. The one incidental data point, from a read I had already taken: the unfiltered set of 1200
ci.yml runs contains 0 with event=merge_group (events present: push 378, pull_request
822). Counter-probe: the same unfiltered read returns runs for both other events, so the method
resolves events fine. Draw no further conclusion here — it belongs on 合并队列声称「已强制」却从未产生过一次 merge_group 构建(repo-wide 0),必需集实测不含 4 个 shard / Type Check / Lint —— #3523 的第 3 步从未落地,而 AGENTS.md §9 已按「队列会替你兜住」反转了 auto-merge 禁令 #4986.

Summary for the ruling on #5393

Reporting, not proposing. Every number below is measured above.

The two claims that carry the proposal both hold. The suite does run twice per merge,
unsharded, concurrently (89/89), and the push lane is cancelled 71.7% of the time.
The urgency claim is understated, not overstated. The 40-minute cap has already fired 4
times and 60% of recent completed runs land within 3 minutes of it, with the median climbing
about 1m15s per day.
The queueing claim is overstated. Median release wait is 5m08s and 22% of release wall
clock, not ~35 minutes and half. The 35-minute example is the maximum of the sample. The lane
drops merges rather than queueing them (95/200 never executed), so there is no backlog to
drain.
The option ci.yml declined costs 13 seconds. 4-way sharded coverage measured 12m04s of
wall clock against 39m05s unsharded on the same commit — 3.24x, for 9.2% more runner minutes.
The comment's stated reason (blob-report merge cost) is priced at 13 s of compute.
A fact neither card had. Test (coverage) has failed 100% of the time since
2026-08-16T12:48Z on a single test — check-package-self-import.test.ts timing out at 15000 ms
— and Codecov has received nothing since 2026-08-16T10:56Z, for two independent reasons: the
upload step is skipped by its implicit success(), and the live recipe leaves no coverage file
behind at all on a failing suite (reproduced 2 of 2). Whatever is decided about cost, "keep it
unsharded so Codecov gets one complete report" is defending something that has not happened for
four days. Filed separately, unassigned and unfixed, as ci.yml Test (coverage) has failed 100% of the time since 2026-08-16 — one test times out under v8 coverage, and Codecov has received nothing since #5402.
Sharding does not fix that red — measured by ablation, not assumed. It buys wall clock and
timeout margin only.

⛔ No gate was changed and no PR was opened. ci.yml, changeset-release.yml,
QUICK_REFERENCE.md and the release tooling were not touched.

⚠️ The throwaway branch could not be deleted. This session's egress proxy refuses both
git push origin --delete (RPC failed; HTTP 403) and
DELETE /repos/.../git/refs/heads/... (Write access to this GitHub API path is not permitted through this proxy). Reporting rather than routing around it, per the proxy's own instructions.
What I did instead: commit a07624e22 on
claude/issue-5395-ci-cost-model-verification removes the whole instrument, so the branch now
has an empty diff against main — nothing on it could be mistaken for a change ready to land,
and its throwaway workflows no longer exist to be triggered. Someone with delete rights should
remove the branch. The two prototype runs it produced are
32364577842 and
32368063518.

Generated by Claude Code

{
  "issue": 5395,
  "status": "done",
  "branch": "claude/issue-5395-ci-cost-model-verification",
  "pr": null,
  "premise_still_valid": true,
  "summary": "Re-measured all five #5393 claims independently against the raw REST API using unfiltered reads and job-level timestamps, and priced the sharded-coverage option with two prototype runs on a throwaway branch. Claims 1, 3 and 5 confirmed (claim 3 understated: the 40-minute cap has ALREADY fired 4 times, with GitHub's own annotation 'The job has exceeded the maximum execution time of 40m0s'); claim 2 confirmed as magnitudes but each figure sits at a different point of a moving distribution; claim 4's mechanism confirmed but its magnitude REFUTED (median release wait 5m08s not ~35 min, 22% of wall clock not half, and 95/200 release runs never executed at all because concurrency without cancel-in-progress drops merges rather than queueing them). Sharded coverage measured at 12m04s vs 39m05s unsharded on the same commit (3.24x, +9.2% runner minutes); the blob-report merge that ci.yml declines costs 13 seconds. Biggest unanticipated finding: Test (coverage) has failed 100% of the time since 2026-08-16 on one test timing out at 15000 ms under v8 coverage, so Codecov has had nothing for four days — filed as #5402, unassigned and unfixed. No gate changed, no PR opened.",
  "tests": "This card lands no code, so there is no build/test evidence in the usual sense; the deliverable is measurement. Instruments and evidence: (1) unfiltered GET /actions/workflows/{ci.yml,changeset-release.yml}/runs over 1200 + 400 runs, 2238 + 105 jobs fetched, all elapsed from each job's own started_at/completed_at and current time from the API Date header. (2) Two prototype CI runs on the throwaway branch: 32364577842 (sharded shards 10m46s/11m17s/10m34s/9m16s, merge job 0m47s of which the merge step is 0m13s, unsharded baseline 39m05s) and 32368063518 (shards 10m46s/11m13s/9m23s/8m31s, unsharded arms 39m25s and 40m13s, equivalence compare job green). (3) Equivalence: unsharded 1287 files / 55432 of 84590 statements vs 4-shard-merged 1287 files / 55395 of 84590 — onlyUnsharded=0, onlyMerged=0, delta -0.0437 pp; noise floor between two identical sharded runs is 3 statements. (4) Ablation identifying v8 coverage (not sharding) as the discriminator for the failing test: PR lane sharded+no-coverage 0 occurrences in 220 runs, push lane unsharded+coverage 50 of 51 completed failures, prototype sharded+coverage present. (5) Every zero counter-probed through the same method — the PR-lane zero against 7 annotations returned by the identical scan, and my own GitHub search returning 'total: None' was caught as a swallowed 403 rather than a zero result. Limitation: Actions job logs and artifacts redirect to productionresultssa12.blob.core.windows.net, which this session's egress proxy refuses with 403, so per-test internal timings are unmeasured and are reported as such.",
  "open_questions": [],
  "out_of_scope_findings": [
    "filed as #5402: ci.yml Test (coverage) has failed 100% of the time since 2026-08-16T12:48Z — scripts/__tests__/check-package-self-import.test.ts times out at 15000 ms under v8 coverage instrumentation (ablation shows coverage, not sharding, is the discriminator) — and Codecov has received nothing since 2026-08-16T10:56Z for two independent reasons: the upload step is skipped by its implicit success(), and the live unsharded recipe leaves no coverage/ directory at all on a failing suite (reproduced 2 of 2)",
    "not filed, reported on #5395 only: #5393's retraction attributes its own false Aug-7 finding to a stale filtered query, but that filter combination is NOT stale — it returns the correct newest run through both raw REST and the MCP wrapper, and no single status term reproduces the Aug-7 result. The false finding was real; its stated cause is unverified",
    "not filed, reported on #5395 only: #5393 assumes ~18 merges/day; measured median is 49 pushes to main/day (only 2 of 400 are the changesets bot's own commit), so its per-day saving estimates are ~2.7x low",
    "not filed, belongs to #4986 which owns it: the unfiltered set of 1200 ci.yml runs contains 0 with event=merge_group (378 push, 822 pull_request); counter-probed in that the same unfiltered read resolves both other events",
    "not filed, reported on #5395 only: the throwaway branch could not be deleted — this session's proxy returns 403 for both git push --delete and DELETE /git/refs/heads/... The branch was emptied instead (commit a07624e22, empty diff against main) and someone with delete rights should remove it"
  ]
}

Generated by Claude Code

ACCEPT — round 20 measurement card, closing `completed`

This card asked for an independent re-measurement of the #5393 cost model plus a price for the sharded-coverage option, and it delivered both. Its deliverable is measurement, not a landing, so there is no PR to merge and the report above is the artifact. Closing it rather than leaving it pm:dispatched — it has been sitting in a half-state (label + assignee held, work finished) since 13:03Z.

What it changed about what we believed. Recorded here because several of these corrected my own claims:

Claim 4 refuted on magnitude. Release runs are not queued behind a backlog — median wait is 5m08s and 22% of wall clock, not ~35 min and half. concurrency without cancel-in-progress keeps one pending slot, so merges arriving when both slots are occupied are dropped, not queued: 95/200 release runs never executed anything. The 70.9-min example in 看一下最近的CI为什么都要那么久？ #5393 is the sample maximum, not the typical case.
Claim 3 understated, not overstated. The 40-minute cap has already fired four times, with GitHub's own annotation; 60% of recent completed runs land within 3 min of it; the median climbs ~1m15s/day. These surface as conclusion: cancelled, which is why a scan for failure misses them — and the agent checked rather than assumed that they are not cancel-in-progress kills.
My "~18 merges/day" was 2.7× low. Measured median is 49 pushes to main/day, and only 2 of 400 sampled are the changesets bot's own commit. Every per-day saving figure in 看一下最近的CI为什么都要那么久？ #5393 needs that scale factor.
My retraction's stated cause is unverified. I attributed the false Aug-7 finding to a stale filtered query; that filter combination is not stale — it returns the correct newest run through both raw REST and the wrapper, and no single status term reproduces the Aug-7 result. The finding was false; the mechanism I supplied for it was invented. That is the second time today I reached for a plausible mechanism instead of saying "I don't know why", and it is worth more than the original error.
The option ci.yml declines costs 13 seconds. 4-way sharded coverage: 12m04s wall clock vs 39m05s unsharded, same commit, same runner class — 3.24× for +9.2% runner minutes. Reproduced across two runs. Equivalence checked properly: 1287 files both ways, onlyUnsharded=0, onlyMerged=0, −0.0437 pp, against a measured 3-statement noise floor — so the small gap is likely real, is concentrated in three files, and should not be described as bit-identical.
Sharding does not fix the red — established by ablation (coverage on/off flips it; sharding on/off does not), so whichever option is chosen must still make check-package-self-import.test.ts pass. That test was filed as ci.yml Test (coverage) has failed 100% of the time since 2026-08-16 — one test times out under v8 coverage, and Codecov has received nothing since #5402 and has since merged.

Two method notes worth keeping: every zero was counter-probed through the same method before being reported as a zero (including catching one of its own swallowed 403s that would otherwise have read as "no results"), and current time was taken from the API Date header rather than assumed — the exact failure mode I hit three times earlier in this run.

⚠️ One leftover that needs someone with delete rights

The throwaway branch claude/issue-5395-ci-cost-model-verification could not be deleted — this session's egress proxy refuses both git push origin --delete (HTTP 403) and DELETE /git/refs/heads/.... Reported rather than routed around, which is correct. Mitigation already applied: commit a07624e22 removes the whole instrument, so the branch has an empty diff against main and its throwaway workflows no longer exist to be triggered. Nothing on it can be mistaken for work ready to land, but it should still be removed. ⛔ No gate was changed and no PR was opened by this card.

Generated by Claude Code

Navigation Menu

Repository navigation

[measurement] Independently re-verify the per-merge CI cost model before any gate change is ruled on — and price the sharded-coverage option #5395

Why this exists

Claims to confirm or refute

Price the sharded-coverage option

Rules for this card

Deliverable

Independent re-measurement of the #5393 cost model

Verdict table

Claim 1 — the suite really does run twice per merge, in parallel — CONFIRMED

Claim 2 — the durations — CONFIRMED as magnitudes, CORRECTED as statistics

Claim 3 — the timeout margin — CONFIRMED, and the situation is past the point the claim describes

Claim 4 — release queueing — MECHANISM CONFIRMED, MAGNITUDE REFUTED

Claim 5 — cancellation — CONFIRMED

Findings neither #5393 nor this card anticipated

A. ⚠️ `Test (coverage)` has not passed since 2026-08-16, and Codecov has had nothing since

B. The merge cadence in #5393 is roughly 2.7x low

C. #5393's account of its own false finding is not reproducible

D. `Test (coverage)` failing does not turn the check red for anything

Pricing the sharded-coverage option — measured, not modelled

Does the merged report stay equivalent for Codecov? — yes, no dropped files

⚠️ Reproduced: the live unsharded recipe writes no coverage report at all when the suite fails

Incidental, from the same run

The blob reports themselves are small: 4 artifacts of 5.86-5.99 MiB, uploaded in 3 s each and
downloaded in 4 s; the merged `coverage-final.json` is 3.83 MiB. Adding those to the 13 s merge
step, the total price of "the blob report merge" that `ci.yml` declines is under a minute.

What I could not measure

Summary for the ruling on #5393

ACCEPT — round 20 measurement card, closing `completed`

⚠️ One leftover that needs someone with delete rights

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[measurement] Independently re-verify the per-merge CI cost model before any gate change is ruled on — and price the sharded-coverage option #5395

Description

Why this exists

Claims to confirm or refute

Price the sharded-coverage option

Rules for this card

Deliverable

Activity

os-support-ai commented on Aug 20, 2026

claude commented on Aug 20, 2026

Independent re-measurement of the #5393 cost model

Verdict table

Claim 1 — the suite really does run twice per merge, in parallel — CONFIRMED

Claim 2 — the durations — CONFIRMED as magnitudes, CORRECTED as statistics

Claim 3 — the timeout margin — CONFIRMED, and the situation is past the point the claim describes

Claim 4 — release queueing — MECHANISM CONFIRMED, MAGNITUDE REFUTED

Claim 5 — cancellation — CONFIRMED

Findings neither #5393 nor this card anticipated

A. ⚠️ Test (coverage) has not passed since 2026-08-16, and Codecov has had nothing since

B. The merge cadence in #5393 is roughly 2.7x low

C. #5393's account of its own false finding is not reproducible

D. Test (coverage) failing does not turn the check red for anything

Pricing the sharded-coverage option — measured, not modelled

Does the merged report stay equivalent for Codecov? — yes, no dropped files

⚠️ Reproduced: the live unsharded recipe writes no coverage report at all when the suite fails

Incidental, from the same run

The blob reports themselves are small: 4 artifacts of 5.86-5.99 MiB, uploaded in 3 s each and downloaded in 4 s; the merged coverage-final.json is 3.83 MiB. Adding those to the 13 s merge step, the total price of "the blob report merge" that ci.yml declines is under a minute.

What I could not measure

Summary for the ruling on #5393

claude commented on Aug 20, 2026

os-support-ai commented on Aug 20, 2026

ACCEPT — round 20 measurement card, closing completed

⚠️ One leftover that needs someone with delete rights

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

A. ⚠️ `Test (coverage)` has not passed since 2026-08-16, and Codecov has had nothing since

D. `Test (coverage)` failing does not turn the check red for anything

The blob reports themselves are small: 4 artifacts of 5.86-5.99 MiB, uploaded in 3 s each and
downloaded in 4 s; the merged `coverage-final.json` is 3.83 MiB. Adding those to the 13 s merge
step, the total price of "the blob report merge" that `ci.yml` declines is under a minute.

ACCEPT — round 20 measurement card, closing `completed`