Describe the bug
On GitHub-hosted windows-2022, about 5% of our Windows test jobs stop communicating mid-step and are reaped 2700 seconds after the last lease renewal, with the annotation "The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." 146 of 3470 jobs since 2026-07-12, flat across the month. A rerun of the same commit passes.
Two parts of this belong to the runner application rather than to our workflow. Nothing survives the death: the log blob is BlobNotFound, the upload steps never run, and _diag goes with the VM, so the agent's last words are unreachable. And the job burns 45 to 70 minutes before the service gives up, while the step's own timeout-minutes never fires because nothing is left to enforce it.
The agent is also not being singled out. Our test step runs a separate PowerShell process that posts a commit status every 15 seconds over its own HTTPS connection. Its last status lands inside the same 30-second window as the agent's last lease renewal, after a cadence of 31, 31, 31, 31, 30 seconds, with process count, handle count and free disk all flat. Whatever happens takes the whole machine at once, from a healthy state, with no ramp.
To Reproduce
Not reproducible on demand at 5.1% per job. It needs a long Windows job: ours is a 9 minute suite of ~240 bash tests driving a native binary against loopback Python fixture servers, 16-way parallel under MSYS2 bash. All 26 deaths with a surviving step record died inside that step. The MSBuild and vcpkg build phase has 350 runner-hours and zero deaths.
Expected behavior
The job fails in less than 2700 seconds when the runner stops heartbeating, or the step's timeout-minutes still applies. Failing that, something diagnostic survives: the partial log blob, or a supported way to stream _diag off the machine.
Runner Version and Platform
GitHub-hosted windows-2022, not -latest. Two runner versions cover the window: 2.335.1 through 2026-07-22, then 2.336.0 from 2026-07-23 to now. Both carry the same rate, 27 of 664 jobs against 119 of 2808 counting every Windows job we ran, z = -0.20, so the upgrade moved nothing. Image generations are flat as well: five over the window, chi-square 3.20 on 3 df, p = 0.36. Killed jobs keep no log, so both versions were read from the setup step of jobs that survived on the same days.
What's not working?
Recent examples, all in the public repo xroche/httrack:
- job 94102074660, runner "GitHub Actions 1000050296", started 2026-08-12T11:44:59Z, last sign of life 11:49:48Z
- job 94083785133, runner "GitHub Actions 1000049912", started 2026-08-12T10:24:31Z, last sign of life 10:29:58Z
- job 94054648381, runner "GitHub Actions 1000049412", started 2026-08-12T08:23:18Z
Measured and excluded from our side: cross-repo runner concurrency, the image generation, ephemeral ports (~2000 connections a run, peaking near 115/s), memory (~1 GB of 16), handles, disk, and our own process kills. On that last one, taskkill /F /T declines a stale-PPID claimant 30 times out of 30 when tested on a real runner, so a tree kill of ours cannot reach the agent through a recycled parent PID. The hazard is per job rather than per runner-hour: fitting kills ~ runs * L^b over eras spanning 4.8 to 21.6 median job minutes gives b = -0.10 [-0.49, 0.28] and rejects b = 1 at p ~ 1e-8.
Job Log Output
None survives. The only trace is the commit status our own watchdog posts every 15 seconds:
t=364s q=174s 0 95_local-sitemap.test | p=136 h=43254 d=84 l=768 x=0 n=7 f=0 e=8 m=2512 c=1453 a=44
Elapsed, seconds since the progress log last moved, test in flight, process count, summed handles, free GB, peak loop lag in ms, failed posts since one landed, TCP connections initiated and failed in the period and currently open, and working-set / paged / agent memory in MB. On a job that dies, these are flat to the last tick.
Runner and Worker's Diagnostic Logs
Not obtainable: _diag lives on the hosted VM, which is destroyed. If there is a supported way to stream those logs off a hosted runner while the job runs, we will add it and come back with the agent's last lines.
Describe the bug
On GitHub-hosted
windows-2022, about 5% of our Windows test jobs stop communicating mid-step and are reaped 2700 seconds after the last lease renewal, with the annotation "The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error." 146 of 3470 jobs since 2026-07-12, flat across the month. A rerun of the same commit passes.Two parts of this belong to the runner application rather than to our workflow. Nothing survives the death: the log blob is
BlobNotFound, the upload steps never run, and_diaggoes with the VM, so the agent's last words are unreachable. And the job burns 45 to 70 minutes before the service gives up, while the step's owntimeout-minutesnever fires because nothing is left to enforce it.The agent is also not being singled out. Our test step runs a separate PowerShell process that posts a commit status every 15 seconds over its own HTTPS connection. Its last status lands inside the same 30-second window as the agent's last lease renewal, after a cadence of 31, 31, 31, 31, 30 seconds, with process count, handle count and free disk all flat. Whatever happens takes the whole machine at once, from a healthy state, with no ramp.
To Reproduce
Not reproducible on demand at 5.1% per job. It needs a long Windows job: ours is a 9 minute suite of ~240 bash tests driving a native binary against loopback Python fixture servers, 16-way parallel under MSYS2 bash. All 26 deaths with a surviving step record died inside that step. The MSBuild and vcpkg build phase has 350 runner-hours and zero deaths.
Expected behavior
The job fails in less than 2700 seconds when the runner stops heartbeating, or the step's
timeout-minutesstill applies. Failing that, something diagnostic survives: the partial log blob, or a supported way to stream_diagoff the machine.Runner Version and Platform
GitHub-hosted
windows-2022, not-latest. Two runner versions cover the window: 2.335.1 through 2026-07-22, then 2.336.0 from 2026-07-23 to now. Both carry the same rate, 27 of 664 jobs against 119 of 2808 counting every Windows job we ran, z = -0.20, so the upgrade moved nothing. Image generations are flat as well: five over the window, chi-square 3.20 on 3 df, p = 0.36. Killed jobs keep no log, so both versions were read from the setup step of jobs that survived on the same days.What's not working?
Recent examples, all in the public repo xroche/httrack:
Measured and excluded from our side: cross-repo runner concurrency, the image generation, ephemeral ports (~2000 connections a run, peaking near 115/s), memory (~1 GB of 16), handles, disk, and our own process kills. On that last one,
taskkill /F /Tdeclines a stale-PPID claimant 30 times out of 30 when tested on a real runner, so a tree kill of ours cannot reach the agent through a recycled parent PID. The hazard is per job rather than per runner-hour: fittingkills ~ runs * L^bover eras spanning 4.8 to 21.6 median job minutes gives b = -0.10 [-0.49, 0.28] and rejects b = 1 at p ~ 1e-8.Job Log Output
None survives. The only trace is the commit status our own watchdog posts every 15 seconds:
Elapsed, seconds since the progress log last moved, test in flight, process count, summed handles, free GB, peak loop lag in ms, failed posts since one landed, TCP connections initiated and failed in the period and currently open, and working-set / paged / agent memory in MB. On a job that dies, these are flat to the last tick.
Runner and Worker's Diagnostic Logs
Not obtainable:
_diaglives on the hosted VM, which is destroyed. If there is a supported way to stream those logs off a hosted runner while the job runs, we will add it and come back with the agent's last lines.