📈 CI Daily Pulse

[radical]

🔴 rolling main 13% — red tip · 🔴 outerloop 0/1 · 🟢 internal recovered (67%) · 🟢 PR 76% — 1 item for a dev

🔴 For a dev — rolling main is red on a recurring Windows VS Code E2E teardown failure

What: The dynamic-debug VS Code E2E cleanup is intermittent, not deterministic: Windows hit EBUSY while removing its temporary workspace in 3 of the last 5 checked rolling runs, with 2 passes between them.
Do: A dev should harden the cleanup at dynamicDebugConfiguration.e2e.test.ts:45 in the failed run; rolling main has no auto-rerun.
_{The same run also had one new Deno breakpoint timeout and one hosted-runner disconnect; internal and PR tips are green — see § 🧩 below.}

_{Window: Thu Sep 3 11:34 → Fri Sep 4 23:34 UTC (last 36h) · deltas vs 7-day average · Sep 4, 2026}

📊 Last 36h by lane — activity, pass rate & Δ vs 7d

Lane	36h activity	36h pass rate (Δ vs 7d)	Latest build
🌳 GH CI — main	🟩🟥🟥🟥🟥🟥🟥	🔴 13% (2/15) · ▼ 18.5	🔴 Sep 4 19:35
🧪 GH CI outerloop — main	🟥🟥	🔴 0% (0/1, 95% CI 0–79%) · ▬ 0.0	🔴 Sep 4 02:11
🏗️ Internal (AzDO) — main	🟩🟩🟩🟩🟥🟥	🔴 67% (6/9, 95% CI 35–88%) · ▼ 6.7	🟢 Sep 4 22:05
🔀 GH CI — PR validation ⭐	🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟥🟥🟥	🟢 76% (106/140) · ▼ 3.3	🟢 Sep 4 23:13

🧱 infra share: unavailable this run. Rolling main's red tip combines a recurring Windows test-cleanup lock, one fresh debugger timeout, and one hosted-runner disconnect; internal AzDO main has 3 failed builds in-window (latest failed 3066502, aggregate counts only) and has since produced three green builds. GH release/13.5 and internal release/13.5 had no runs in the 36h window, so their rows are omitted.

_{bar length = sample size / confidence (short = read its % with care) · Δ vs 7d = 36h rate minus the lane's 7-day average · Latest build = red/green of the lane's most recent in-window run, linked to it · lane names link to the workflow / pipeline.}

🧩 What's broken & why

One thing needs a human: rolling main is red at the tip on a recurring Windows VS Code E2E teardown failure, and rolling has no rerun bot. Internal and PR validation recovered green at the tip; outerloop has been quiet since its one failed nightly run.

	What's broken	Who's on it	Since
🔴	Rolling main is red at the tip: its recurring Windows test-cleanup lock struck again, alongside one new debugger timeout and one lost hosted runner (run).	Needs a dev to triage — no auto-rerun on rolling	recurred ~4h ago; still red
🟡	Internal (AzDO) main is at 67% after 3 of 9 builds failed, but the tip is green after three consecutive recoveries; aggregate counts do not expose a cause here.	Watch — recovered at the tip; cause unconfirmed this run	latest failure ~8h ago; green since
🟡	PR validation pooled 76% over the window, but its tip is green and failures remain scattered rather than one common break.	Rerun bot is absorbing these (PR-only)	ongoing; green at the tip
⚪	Scheduled outerloop main failed its only in-window run across its own CLI and Dashboard suites after all four attempts, with no newer scheduled run yet.	Looks quiet — next nightly run is the confirmation	last failed ~21h ago

⏱️ Rolling main is also running slow today: build-time p50 is ~52m vs ~34m over 7 days.

🧱 Product vs infra: the actionable rolling failure is a recurring test-harness cleanup lock, not a product regression; the same run's debugger timeout is too new to classify, its lost runner is environmental, and the internal cause is unavailable under the counts-only policy.

exact tests, jobs & counts

🔴 Rolling GH CI — main — 36h window (15 counted runs: 2 success, 13 failure; cancelled / in-progress excluded).

Current red tip 33911967287 (Sep 4 19:35):

• VS Code extension E2E (Windows, dynamic-debug-configuration) failed during teardown after the test completed: AggregateError: Dynamic debug configuration E2E teardown failed.
• The exact cleanup error was Error: EBUSY: resource busy or locked, rmdir 'C:\Users\runneradmin\AppData\Local\Temp\aev-lj1QWh\workspace\.e2e-dynamic-debug\second'.
• The cleanup call is at dynamicDebugConfiguration.e2e.test.ts:45.
• The same EBUSY teardown failure occurred in 33887016795, 33910459971, and 33911967287; the matching job passed in 33881358633 and 33902845984. That is 3 of 5 checked runs, confirming intermittent recurrence.
• VS Code extension E2E (Linux, deno-debugger) timed out after 240 seconds waiting to hit its TypeScript breakpoint at denoDebugger.e2e.test.ts:65-72. This is one new signal, not yet a pattern.
• Aspire.Hosting.Java.Tests on Windows lost communication with its hosted runner. GitHub's check annotation identifies the runner disconnect.

Successes: 33791618334, 33881358633.
Failures: 33782779276, 33789740828, 33802459402, 33807224647, 33811778450, 33814653161, 33821649109, 33871922580, 33879357897, 33887016795, 33902845984, 33910459971, 33911967287.

🟡 Internal (AzDO) main — 9 in-window builds: 6 partiallySucceeded (= success), 3 failed (3065629, 3066439, 3066502). Builds 3066559, 3066777, and latest 3066882 recovered to partiallySucceeded. Aggregate counts only per internal-privacy policy; no job names or logs.

🟡 PR validation — pooled 106/140 = 76%; latest run 33928696184 is green. Failures are scattered across jobs with no directly confirmed dominant cluster; the auto-rerun bot rescued 28 red→green.

⚪ Scheduled outerloop main — 1 counted run in-window (33828588223, Sep 4 02:11), failed at attempt 4. Failing suites: Cli (Ubuntu + Windows), Cli.EndToEnd (Ubuntu), and Dashboard (Ubuntu + Windows).

_{Attention dots: 🔴 needs a human now · 🟡 keep an eye on it · ⚪ looks resolved. These are distinct from the lane table's 🔴/🟢 run-outcome dots.}

🔁 Reruns & flaky tax

The auto-rerun bot carried most of the load (112 bot reruns and 28 rescues vs 26 human reruns and 11 human-only rescues), so PR flakiness is mostly absorbed automatically; rolling main has no equivalent safety net.

rerun actor split (36h)

🤖 Auto-rerun bot: 112 reruns across 58 PR runs → 28 rescued red→green; 22 still red, 8 cancelled.
🧑 Manual (human) reruns: 26 reruns across 19 PR runs (1 branches the bot also reran) → on the 17 human-only branches: 11 rescued, 5 still red, 1 cancelled.
🔗 Combined: 77 distinct PR runs needed a rerun in the window (41 went green, 27 still red, 9 cancelled); 1 PR branches needed both bot + human.

_{Sources: gh api .../ci.yml/runs (pass rates + rerun split) · az pipelines build list def 1602 (internal, counts only) · failed GitHub job logs and check annotations for live-cause confirmation. ciinsights could not be enabled in this execution environment, so cluster and infra-share joins could not be refreshed. Window = last 36h. Backlog & trend: see the weekly CI Health report (#18231). Generated Sep 4, 2026 23:34 UTC.}

[joperezr]

🤖 Aspire triage pass — 2026-06-16 06:54 UTC

Area: �rea-engineering-systems
Kind: docs (informational CI health report — [pipeline trial] per title)
Repro: n/a — this is a generated dashboard, not a bug report
Affected version(s): n/a (CI health snapshot, window: last 36h ending 2026-06-16)
Severity: S4 — informational; no product regression. Body flags real CI breakage (SqlServer conformance cluster, outerloop on main red 7+ days), but those are tracked as separate test-failure issues.

Urgency: ⚪ backlog — no milestone
Reason: informational dashboard / pipeline trial output, not a fix candidate. CI health concerns it surfaces are already being tracked under �rea-engineering-systems work.

Suggested owner(s): Jose Perez Rodriguez (@joperezr) (score 6)
Evidence:

• joperezr merged Merge release/13.4 to main after v13.4.0 release #17817, Merge release/13.3 to main after v13.3.5 release (supersedes #17337) #17343 in �rea-engineering-systems (last 90d)
• Issue author (Ankit Jain (@radical)) is the originator of this pipeline trial and is excluded from owner suggestions per policy

Confidence: medium
Needs human?: no — informational; Ankit Jain (@radical) is driving the trial and can dismiss/close when the experiment is over.

Dry-run mode — no labels or milestone applied. Reply /triage apply to promote, or edit and reapply.

[radical]

🔧 Post-run review — 2026-06-18 19:48 UTC

First run of the new two-tier Pulse spec (aspire-ci-pulse.prompt.txt + shared aspire-ci-report-common.prompt.txt, both committed today). The prior body here was produced by the older combined prompt, so a straight diff looks like a big regression — most of it is the intended split, but two things are worth a human's eye.

Regressions vs previous run: none that are true regressions — but two intended discontinuities a watcher will notice:

• Dropped sections (intended): the trend PNG, the <details> per-day grid, and the old "Newly…" flaky block are gone. The new spec explicitly moves trend/grid/build-time to the Weekly (🩺 CI Health — Last 7 days #18231); the Pulse is now current-state-only. Lane labels also moved to the canonical 🌳🌿🧪🏗️🔀 set. Deliberate, not a regression.
• Pass rates jumped sharply (methodology correction, NOT a real swing): PR validation reads 83% (76/92) vs the prior body's 38% (76/200); main reads 30% (6/20) vs 26% (6/23). Same passing counts — the denominator shrank because the new shared counting rule excludes cancelled / in-progress / queued runs, and PR ci.yml has heavy cancellation (65 cancelled in 36h alone). The new numbers are spec-correct; the jump reflects the rule, not reality. Calling this out so nobody reads it as "CI suddenly healed overnight."

Readability / improvement ideas:

• The headline blockquote is dense (lane + 3 tests + rerun split + two healthy lanes + watch clause). Skimmable but near the limit — accept: consider trimming the "PR + internal are healthy" clause to a half-sentence, since the table shows it. Reject if the contrast is the point.
• The activity bars are in a fenced code block so the emoji columns align on every client — reads cleanly. Keep.

Data-gathering issues this run:

• ciinsights query_test_failures threw An error occurred invoking 'query_test_failures'; get_failure_summary worked and supplied the top breakages + infra/flaky counts, so the Pulse was unaffected. The connection status also reported a DB index-creation warning (IX_TestResults_FullyQualifiedName_CapturedAt already exists) — API healthy, but flag if failure data ever looks stale.
• ciinsights test data can lag the last ~30 min (noted in-body); the rerun split is computed live from gh api .../ci.yml/runs, so it's current.
• First --paginate | jq pass silently produced per-page arrays (mangled counts); re-running with --slurp fixed it. The 36h ci.yml window held 194 runs across pages — a non-slurped read would have undercounted.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, infra-share rule. It says NN% (failed-infra/failed-total) / "share of failed builds", but the only figure ciinsights exposes is infrastructureFailureCount vs failingTestCount (test-result level: this run, 77/134 = 57%), not a build-level ratio. Pin the metric to one definition + one ciinsights field so successive runs don't silently switch denominators (build-level vs test-result-level). Small wording fix, durable payoff.

[radical]

🔧 Post-run review — 2026-06-18 20:48 UTC

Regressions vs previous run: none. The diff vs the 19:46 run is entirely genuine state movement — window slid ~1h, MongoDB ReplicaSet overtook the CLI TypeScript-apphost smoke as the single loudest test (11 vs 8), rerun split drifted (56/17 → 55/16), infra share recomputed (77/134 → 78/136), and 7d-avg deltas moved a fraction. No lost structure, no flipped statuses, no format churn. (Also trimmed a few trailing blank lines the prior body had accumulated.)

Readability / improvement ideas:

• (accept/reject) The headline now distinguishes the single loudest test (MongoDB ReplicaSet) from the loudest family (Aspire.Cli.*) in the 👀 Watch clause. This reads better than the prior run, which conflated the two. Suggest keeping this single-test-vs-family split as the headline convention.

Data-gathering issues this run (the valuable one):

• The rerun-split fetch via gh api .../ci.yml/runs?...&created=>2026-06-17... silently truncated to 82 of 169 runs on the first attempt because the > in the created filter was not URL-encoded — the shell/gh mangled the operator and the API ignored the lower bound, returning only one partial slice. I caught it only because I cross-checked against the gh run list count (94 completed PR runs in 36h) and against the API's own total_count (169). Without that reconcile, the rerun split would have been undercounted by ~50% (≈27 bot reruns instead of 55), badly distorting the flaky-tax section.
  • Fix that worked: URL-encode the operator (created=%3E2026-06-17T08%3A38%3A49Z) and --paginate, then assert jq -s length == .total_count.

Suggested prompt change (for a human to apply):

• File aspire-ci-report-common.prompt.txt, "Reruns & the flaky tax" section (or the "Fetch-enough-then-filter" section, which today only warns about gh run list --limit undercounts, not the runs-API created filter). Add one line: "When using gh api .../ci.yml/runs with a created=>… bound, URL-encode the operator (%3E) and verify the fetched run count equals the response's total_count — an unencoded > is silently dropped and truncates the window, halving the rerun split." This is a recurring trap (it will bite every run that fetches the rerun split this way), so it clears the "helps every future run" bar.
• Minor: the prompt never states the infra-share denominator. I inferred infra / (infra + failing) from the prior run's 77/134; this run's 78/136 matches the same formula. Worth one explicit line in the "Infra-share line" section so the figure is reproducible and doesn't drift between runs/authors.

[radical]

🔧 Post-run review — 2026-06-18 21:48 UTC

Regressions vs previous run: none. The diff vs the 20:46 run is entirely genuine state movement — window slid ~1h; MongoDB ReplicaSet ticked 11→12; the CLI TypeScript-apphost smoke (LatestCliCanStartStableChannelTypeScriptAppHost) aged out of the 36h window and StartAsync_…GracefulSignaler… took the #3 slot; rerun split drifted (55/16 → 58/20); infra share recomputed (78/136 → 79/135); 7d-avg deltas moved a fraction as the baseline window also slid. No lost structure, no flipped statuses, no format churn.

Readability / improvement ideas: none new — the headline + first table still convey "what's on fire" without scrolling.

Data-gathering issues this run (the new one):

• ciinsights.query_test_failures failed outright this run (An error occurred invoking 'query_test_failures'), and get_connection_status reported the backend DB as unhealthy (index ... 'IX_TestResults_FullyQualifiedName_CapturedAt' already exists — a stuck server-side migration). I fell back to ciinsights.get_failure_summary (its topFailingTests array) for the §3 breakages list, which worked. Without that fallback the loudest-breakages section would have had no source.
• This is plausibly recurring while the DB stays in that migration state, so it clears the "helps every future run" bar.
• I avoided the prior run's gh api .../ci.yml/runs created=> truncation trap by fetching paginated pages (per_page=100&page=N, 6 pages) and filtering by timestamp in code instead of an API-side created bound — confirms that prior finding's value.

Suggested prompt change (for a human to apply):

• (new — common file, "Reruns & the flaky tax" / data-sources) Add one line: "If ciinsights.query_test_failures errors, fall back to get_failure_summary.topFailingTests for the loudest-breakages list (same fail-count ranking, narrower window control)." Codifies a working mitigation for a tool that was down this run.
• (carried forward from 20:46, still unapplied — restated here because that comment is now resolved) (a) "Fetch-enough-then-filter": warn that an unencoded > in a runs-API created=>… bound is silently dropped and halves the rerun split — URL-encode (%3E) and assert fetched count == total_count, or fetch by page and filter in code. (b) "Infra-share line": state the denominator explicitly — infra / (infra + failing) (this run 79/135) — so the figure is reproducible and doesn't drift between authors.

[radical]

🔧 Post-run review — 2026-06-18 22:52 UTC

Regressions vs previous run: none. Diff against the prior body (published ~1h earlier) is all genuine state movement — window slid forward ~1h, main 6/20→6/21, internal main 14/16→14/17, PR validation 82/98→89/104, latest failed internal build 3003323→3003488, rerun counts 58/32→59/33. No section, column, link, or legend was dropped; no frozen value moved (the Pulse has no frozen history); no status emoji flipped without a real change.

Readability / improvement ideas:

• (accept/reject) The Watch clause was sharpened from last run's broad Aspire.Cli.* to the specific graceful-shutdown Signaler…Escalates family (3 named tests, repeating). This is more actionable — a reader can grep three exact test names. Suggest keeping Watch clauses this specific going forward rather than naming a whole namespace.
• The report is skimmable in ~15s: headline → lane table → reruns → breakages. No change needed.

Data-gathering issues this run:

• recentTrends.last24h.newFailingTests is a names-only list with NO per-test counts, and a large block in it is often ONE build's worth of a whole test class failing once — not N independent signals. This run it listed ~33 DistributedApplicationPipelineTests.* entries; cross-checking via search_tests showed every one had totalRuns=1, failures=1, all at the same timestamp (2026-06-17T23:03:34) — a single-build setup/compile blip, not a repeating breakage. Had I trusted the raw block I'd have featured a phantom "loud new family." Always validate newFailingTests entries against search_tests/failCount before featuring them.
• PR-validation 7d denominator needed a much larger fetch than the common prompt's --limit 300. At ~90–150 runs/day, 300 runs only spanned ~2.5 days (back to Jun 16) — short of the 7-day window. Refetched at --limit 1200 (reached Jun 7). The "fetch-enough-then-filter" rule held, but the suggested 300 floor silently undercounts this lane's 7d average.
• ciinsights get_connection_status reported a DB index warning (IX_TestResults_FullyQualifiedName_CapturedAt already exists); all queries still returned complete data. One-off, no impact this run.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, the "Fetch-enough-then-filter" section: add a note that for the 7-day average on PR validation specifically, --limit 300 is insufficient (covers only ~2–3 days); use --limit ~1200 (or paginate to the 7d start) for that lane. The 36h window is fine at 300.
• aspire-ci-report-common.prompt.txt (or the Pulse §3 / headline Watch guidance): add a caution that recentTrends.newFailingTests carries no counts and a bulk block is frequently a single failed run of one test class — verify against search_tests before treating it as a flaky/broken family or featuring it in the Watch clause.

[radical]

🔧 Post-run review — 2026-06-18 23:52 UTC

Regressions vs previous run: none. Diffed the prior body against this one — every section, column, link, the bar-length legend, and both <sub> lines are intact. The only changes are state-driven value updates (window slid ~1h, so the MongoDB ReplicaSet 36h count fell 12→8 as older fails aged out; internal main 82→88%, PR validation 86→85%, infra share 59→60%). Pulse freezes no history, so there is no frozen-cell drift to check.

Readability / improvement ideas:

• (accept/reject) The report can be built from one ciinsights call: get_health_summary returns both failureBreakdown (testFailures + infrastructureFailures + total → the 🧱 infra-share fraction) and topFailingTests (the §3 loudest-breakage list). This run, query_test_failures failed outright (server-side DB index error: IX_TestResults_FullyQualifiedName_CapturedAt already exists), but get_health_summary + get_failure_summary covered everything. Suggest the prompt name get_health_summary as the primary §1-infra/§3-breakages source so a future run doesn't get blocked on the flaky query_test_failures endpoint.

Data-gathering issues this run:

• Rerun split is methodology-sensitive and the prompt under-specifies it. The window moved only ~1h vs the prior run, yet bot reruns jumped 59→89 and human 20→23. Two unpinned choices drive this: (a) what counts as one "rerun" — I used Σ(run_attempt−1) per run, attributed to the latest-attempt actor; (b) how to identify a PR — 38 of 80 rerun-bearing ci.yml runs were fork PRs with an empty pull_requests array, so I fell back to head_branch as the PR-identity key (raising distinct-PR from 23 to 48, much closer to the prior run's 46). Without these pinned, two runs an hour apart can legitimately disagree by ~50%.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, the "Reruns & the flaky tax" section: pin the counting unit and the join key explicitly — (1) a "rerun" = each attempt beyond the first, i.e. Σ(run_attempt−1); (2) attribute a run's reruns to its latest-attempt triggering_actor (github-actions[bot] = bot, else human); (3) when pull_requests is empty (fork PRs), key the PR by head_branch. This is additive — it doesn't contradict the existing "split by who triggered it" rule, just makes it reproducible run-to-run.
• aspire-ci-pulse.prompt.txt / common data sources: note get_health_summary as the single source for both the infra-share fraction and the loudest-breakage list, with query_test_failures flagged as currently erroring.

[radical]

🔧 Post-run review — 2026-06-19 18:52 UTC

Regressions vs previous run: none. All deltas vs the prior body reflect genuine state changes. Structure preserved (header, headline, single lane table, infra line, legend, rerun split, loudest breakages, footer). The PR-validation activity bar shrank 12→11 cells — expected, not a regression: bar length is log-scaled to sample size and the lane's 36h run count dropped (109→72). The Internal AzDO release/13.4 row was omitted correctly (0 builds in window) and the omission is annotated inline so a reader doesn't think the lane was dropped silently.

Readability / improvement ideas:

• (accept/reject) The headline now carries two distinct test clusters (analyzer pair + EF Core Enrich) plus the "loudest by count vs most-recently-active" nuance. It's accurate but dense — at ~5 lines it's near the edge of skimmable. Consider a soft cap: lead with the single loudest cluster, push the "most-recently-active" distinction to §3 only. Reject if the recency caveat is considered load-bearing for "what's on fire now."

Data-gathering issues this run:

• Rerun-split attribution is an approximation. I bucketed reruns by triggering_actor of the latest attempt and counted run_attempt − 1 reruns per run, all assigned to that one actor. A run rerun first by the bot then by a human (or vice-versa) is attributed entirely to whoever triggered last; mixed-actor runs are only partially captured (I could detect "needed both" = 1 only via PR-set overlap, not per-attempt). This likely explains some of the large magnitude swing from the prior run (bot 89/human 23 → bot 43/human 16) — part real drop, part method variance. The prompts do not pin an exact counting method, so two runs can compute this differently.
• Infra-share denominator is ambiguous. The common prompt defines 🧱 infra share = failed-infra/failed-total as "share of failed builds", but ciinsights returns failure-occurrence counts (infrastructureFailureCount vs failingTestCount). I used 62/102 (infra ÷ infra+test signals); the prior run used 83/138 — a different denominator entirely. The metric isn't reproducible run-to-run without pinning the source + denominator.
• ciinsights get_test_history returned empty for an exact FQN (...MSBuildTests.AspireExportAnalyzers...) that get_failure_summary simultaneously reported with 34 fails in the same 36h window. So I could not use it to attribute the loudest test to a specific lane; the headline characterizes the analyzer pair by suite/family from the summary alone, not by lane. get_flagged_tests(problemType=broken) also did not surface the 34-fail pair in its top rows (it ranks multi-context broken tests; the pair appears single-context), so neither tool corroborated a lane for it.
• ciinsights get_connection_status reported the API healthy but the DB "unhealthy" (a duplicate-index warning, IX_TestResults_FullyQualifiedName_CapturedAt). All queries still returned data; flagging in case it foreshadows stale/partial results on a future run.

Suggested prompt change (for a human to apply):

• aspire-ci-report-common.prompt.txt, "Reruns & the flaky tax" section: pin the exact rerun-counting method — e.g. "count reruns as Σ(run_attempt − 1) over completed ci.yml runs in the window; attribute each run's reruns to the triggering_actor of the latest attempt (a known limitation for mixed bot+human runs); bot = login ending [bot]." Without this, the bot-vs-human magnitudes are not comparable across runs.
• Same file, "Infra-share line" section: pin the denominator and source — state whether it's failed builds or failed signals, and that ciinsights failingTestCount / infrastructureFailureCount are the inputs (or specify the build-level alternative). This run (62/102) and the prior (83/138) used different denominators for the same metric.