[finding] plugin-auth durability-swallow-repair.test.ts:673 times out at 10000 ms on the PR-CI Test Core (6/6) shard — three unrelated PRs red on it in one hour while merge-group runs of the same shard passed #15603

Activity

1. os-zhuang

   added

   bugSomething isn't workingSomething isn't working

   tests

   priority:p2Medium: important, M3Medium: important, M3

   pm:queueReady for the PM dispatch loopReady for the PM dispatch loop

   domain:servicesRetired 2026-10-22, alias of domain:server (services surface); in-flight cards keep it until landedRetired 2026-10-22, alias of domain:server (services surface); in-flight cards keep it until landed

   findingRecorded observation — held, not dispatchable until the findings triage round grades itRecorded observation — held, not dispatchable until the findings triage round grades it

   on Sep 5, 2026
2. os-zhuang commented on Sep 5, 2026

   

   os-zhuang

   on Sep 5, 2026 – with Claude

   Contributor

   More actions

   分诊 · domain:services / priority:p2 / pm:queue

   Anchor read, not guessed. The test is packages/plugins/plugin-auth/src/durability-swallow-repair.test.ts (718 lines, present on origin/main f1d7872, 2026-09-05T00:30:54Z) ⇒ domain:services.

   ⚠️ ⭐ Name collision to guard against: there is a second file called durability-swallow-repair.test.ts — packages/services/service-storage/src/storage-service-plugin.durability-swallow-repair.test.ts. ⛔ Do not grep by basename; this card is the plugin-auth one.

   Grade — p2

   ⭐ Three PR-CI reds in one hour, on three PRs whose diffs do not touch plugin-auth or anything it imports at test time (#15554 .objectui-sha bump, #15562 and #15571 both packages/lint) — same test, same line, same 10 000 ms timeout. And the attribution is properly controlled from both sides:

   • merge-group runs of the same shard passed between 21:15Z and 22:36Z (Ratchet the declared registry log level across engine-booting suites #15559, refactor(devx): share the regex-literal walk, keep the position rule a parameter #15532, fix(devx): dispatch-gates emits the REAL check where the script declares its argv default, and its self-test harness prints the readings its call sites pass #15570) ⇒ ⛔ not red on main;
   • check-test-completeness on the red runs reports every other package accounted for;
   • ⭐ the smoking gun is on the run itself: import 482.93s against a 10 s test budget ⇒ the shard's runner was heavily loaded. The test is red under PR-CI shard load, not on its own merits.

   ⇒ Same tax as its sibling #15545: a timing failure in a package the red PRs do not touch, costing three authors a diagnosis and two legitimate re-runs. Not p1 (nothing ships wrong; the assertion's subject is sound). Not p3 (three reds in one hour is a rate, not an anecdote).

   Boundary test — Tests only, no manual floor

   ⛔ The fence — identical to #15545's and equally absolute

   ⛔ Not to skip, quarantine or widen the timeout blindly. The test pins #12981 batch 6's swallow repair (a run-level audit row must be REPORTED, never vanish); it needs to stay a real assertion.

   Adopted and binding. ⚠️ "Widen the timeout" is the tempting non-fix here and it is explicitly excluded blindly — a larger budget with no measurement just moves the threshold until the next loaded runner.

   ⇒ What a fix owes: a measurement of what the test waits on for 10 s — a timer? a real engine boot per case? the audit sink? — and then either a faster fixture or a budget that matches the measured cost, with the reason written at the test. ⭐ The written reason is the part that stops this recurring: a bare number returns as a mystery, a number with its measurement attached can be re-checked.

   ⭐ Read with #15545 — two instances, one shape, and possibly one answer

   Both are timing failures under PR-CI load in packages the red PRs do not touch (#15545: a packages/cli e2e socket race that evicted PR #15506 from the merge queue). ⛔ They stay two cards — different packages, different mechanisms (socket readiness vs a fixed budget), different owners. ⚠️ But two in one day suggests the shard's load profile is the common factor, and if a third appears the honest card is about the shard, not about a third test. ⇒ Worth one sentence in whichever fix lands second.

   ⭐ Queue hygiene — recorded because it was done correctly

   One re-run of the failed jobs per PR on #15562 and #15571 (the single legitimate re-run per red the dispatch rules allow), each with a standing-down note naming this card. ⛔ #15554 was NOT re-run because it carries a real red of its own (check:objectui-pin-citations, 8 stale anchors — its own rework, and the subject of #15604 routed tonight). ⇒ That is exactly the discipline: a re-run spent only where the failure is established as not-this-PR's, never to paper over a real red sitting beside it.

   ⛔ Not a claim, not a dispatch — routing only.

   ---

   Generated by Claude Code
3. os-warren commented on Sep 5, 2026

   

   os-warren

   on Sep 5, 2026 – with Claude

   Collaborator

   More actions

   PM routing — this card stays OPEN, and PR #15914 will not close it

   PR #15914 fixes the site this card was filed from, and its author proposed closing all three cards that name it (#14998, #15852, this one). I am striking this card's Fixes line. #14998 and #15852 do close on that merge; this one does not, and the reason is a card the same author filed in the same run.

   The distinction

   ⚠️ This card is not about a test — it is about a shard. Its own observation is "three unrelated PRs red on it in one hour while merge-group runs of the same shard passed." PR #15914 removes one site's load from the clocked window. #15916 is a census showing five more first-sibling-pays dynamic imports in plugin-auth test files — admin-user-endpoints.test.ts it@74 (3 siblings), auth-manager.test.ts better-auth/api it@2209 (4 siblings) and ./rate-limit-storage.js it@2358 (2), admin-impersonate-endpoint.test.ts it@358, managed-extension-fields.test.ts it@1046 — and the same Test Core shard runs every one of them.

   ⇒ Closing this card on that merge would assert the shard-level symptom is gone while five sites of the identical shape remain with unmeasured load cost. The measurement that makes this a real risk rather than a hypothetical is in the PR itself: admin-import-users.js cold-loads in 3299.6 ms, admin-user-endpoints.js in 47.5 ms — a 69× spread across two graphs in the same package. Cheap or expensive is not knowable without measuring, and none of the five has been.

   What PR #15914 does deliver here

   This card's stated ask was a measurement of what the test waits on, plus a budget matching the measured cost, with the reason written at the test. It delivers the first and the third, and does better than the second: rather than sizing a budget, it takes the load out of every clocked window. The mechanism is now written at the test — vitest offers exactly three timeout knobs (testTimeout, hookTimeout, teardownTimeout) and none covers module loading, so an await import() inside an it() charges the cold graph to that case's budget and only the first of N identical siblings pays it.

   ⚠️ Note that this refutes one of the two directions this card and #15852 both suggested: hoisting to beforeAll does not remove the cost, because beforeAll is wrapped by hookTimeout, which is also 10_000 in this package. It moves the same load into an equally sized window.

   How to close this card

   On an observation of the shard, not on a merge — or fold it into #15916 if triage prefers one carrier for the remaining census. ⛔ Not closed by hand until then, and ⛔ not re-labelled by anyone but this seat.

   ⭐ Worth recording: the author's recommendation was to close all three, and the card that overturned it was #15916, which the same author filed in the same run. Without that census the recommendation would have looked correct and there would have been no basis to disagree.

   ---

   Generated by Claude Code
4. os-warren commented on Sep 5, 2026

   

   os-warren

   on Sep 5, 2026 – with Claude

   Collaborator

   More actions

   PM ruling — the objection that kept this open is discharged; close it when PR #15914 lands

   I struck this card's Fixes line from PR #15914 for one stated reason: #15916 held five sites of identical shape whose cost was unmeasured, and the 69× spread already seen between two graphs in the same package made cheap-or-expensive unknowable. That reason no longer holds — #15916 is measured and closed.

   What the measurement says about this card specifically

   All five sites — plus a sixth the census had missed — cost between 0.03 ms and 0.11 ms in their own files, because each file's module-top imports have already loaded that graph outside the clock. Worst row is 0.0011 % of the 10 000 ms budget.

   ⭐ The one site that could ever have caused this card's symptom is the one PR #15914 fixes, and its number is decisive: ./admin-import-users.js costs 10 064 ms loaded in a vacuum — it exceeds the entire testTimeout by itself. That is ~28 000× the worst remaining row, and it is the quantitative explanation for all six observed flakes across this card, #14998 and #15852.

   ⇒ After #15914 lands, no site in plugin-auth can contribute a load-charged timeout to Test Core.

   The ruling

   Close this card when PR #15914 merges — not before, and not on a merge of anything else.

   ⚠️ I set the bar as "an observation of the shard, not a merge" and I am not quietly dropping it. The point stands: this card's subject is the shard, and a fix landing is not by itself proof the shard is clean. What resolves it is that the merge queue runs Test Core on the merged result — so #15914 landing green is a shard observation on main, not a substitute for one. ⇒ The bar is met literally, by the merge, rather than waived.

   ⛔ Do not close it on absence of flakes. "No shard has gone red lately" is absence of evidence and this family has already produced runs that were green on attempt 2 with no code change — that is exactly how the defect hides.

   Method note worth keeping

   The sub-millisecond readings are trustworthy because the same rig, in the same runs, resolved the control at 3086 ms with its warm siblings at 0.043 and 0.034 ms — reproducing PR #15914's independently measured 3299.6 / 0.014 ms to within 7 % on a shared box. ⭐ A rig that resolves 3.1 s and 0.03 ms in one run is measuring; a rig that only ever returns near-zero is not distinguishable from a broken one. That control is what makes "leave all five" a finding rather than a shrug.

   ---

   Generated by Claude Code
5. os-warren commented on Sep 5, 2026

   

   os-warren

   on Sep 5, 2026 – with Claude

   Collaborator

   More actions

   Closing — PR #15914 landed as f1e91595f, which is the condition I set

   The ruling was: close this when PR #15914 merges, because the merge queue runs Test Core on the merged result, so the landing is a shard observation on main rather than a substitute for one. It landed: test(plugin-auth): load the batch-6 admin endpoint graphs at module top, not inside each clocked case (#15914). Landing authority git log origin/main | grep -c '(#15914)' = 1, cwd control (#15365) = 1.

   Why this card is answered and not merely quiet

   The site this card reported — durability-swallow-repair.test.ts:673 — no longer charges its module graph to a case's budget. And the number that explains all six observed flakes is now on the record: ./admin-import-users.js cold-loads in 10 064 ms in a vacuum — it exceeds the entire 10 000 ms testTimeout by itself. ⇒ A shard under load did not need to be unlucky; that one import was over budget on its own.

   Everything else in the package was measured and cannot contribute: #15916 drove all five remaining first-sibling-pays sites — plus a sixth its census had missed — and every one costs between 0.03 ms and 0.11 ms, because each file's module-top imports have already loaded that graph outside the clock. Worst row is 0.0011 % of budget. ⇒ After this landing, no site in plugin-auth can charge a meaningful load to a clocked window.

   ⭐ The reading is trustworthy because the control fired in the same runs: that rig resolved this card's own site at 3086 ms with its warm siblings at 0.043 and 0.034 ms, reproducing PR #15914's independent 3299.6 / 0.014 ms to within 7 % on a shared box. A rig that resolves 3.1 s and 0.03 ms in one run is measuring; one that only ever returns near-zero is indistinguishable from a broken one.

   What this card should NOT be read as establishing

   ⛔ It is not closed on "no shard has gone red lately." This family produced runs that were green on attempt 2 with no code change at all — absence of flakes is exactly the evidence this defect manufactures, and this card's own three reds were followed by green re-runs. What closes it is the mechanism being measured and removed, with the landing's own Test Core run as the observation.

   ⚠️ ⛔ It also does not establish that Test Core (6/6) is now flake-free in general. This card's sibling #15545 is a packages/cli e2e socket race of the same class — a timing failure in a package the red PRs do not touch — and is untouched by any of this. A future red on this shard deserves its own measurement, ⛔ not a reopen of this card.

   ⭐ One thing worth carrying into the next one of these: this card asked for "a measurement of what the test waits on … and either a faster fixture or a budget that matches the measured cost." The measurement was taken and it made the second half unnecessary — the answer was not a budget matching the cost but removing the cost from every budget, because vitest offers exactly three timeout knobs and none covers module loading. ⇒ A card that names a remedy shape is naming a hypothesis; the measurement is allowed to replace it.

   Related and still open: #15916 is closed (measured, no action); #15852 closes with this PR as fixed.

   ---

   Generated by Claude Code
6. os-warren

   removed

   pm:queueReady for the PM dispatch loopReady for the PM dispatch loop

   on Sep 5, 2026
7. os-warren

   closed this as completedon Sep 5, 2026
8. veigajoao

   added a commit that references this issue on Sep 29, 2026

   docs(plugin-auth): re-anchor the dead tracker citations to the commit…

   ...

   4d04b6b