Repository navigation
New WSL sessions hang under memory pressure with vmbus_alloc_ring allocation failure #40795
Description
AI disclosure
This report was drafted and filed with assistance from OpenAI Codex, based on local command output and logs from the affected machine.
Summary
WSL semi-routinely gets into a state where existing WSL shells continue to work, but new WSL sessions and Windows interop hang. This appears to correlate with memory pressure and a kernel vmbus_alloc_ring allocation failure.
This looks similar to #11612. It has similar outer symptoms to #12991, but Docker Desktop is already newer than the Docker Resource Saver fix discussed there and in docker/for-win#14656.
Environment
- WSL 2
- Distro: Ubuntu 24.04.3 LTS
- Kernel:
6.18.33.1-microsoft-standard-WSL2 .wslconfig:memory=52GB- Docker Desktop:
4.44.3 - Docker Engine / CLI:
28.3.2 - Docker Compose:
v2.39.1-desktop.1 - Docker Desktop WSL bootstrap:
098d711f2ff00f0a1c4d69da1f9d6d2d - Windows / WSL app version: not collected while wedged because
wsl.exe --statusandwsl.exe -l -vhang
Behavior
After some time under normal dev workload, apparently under memory pressure:
- existing WSL shells still run commands
- opening new WSL terminals hangs
- launching
wsl.exefrom WSL hangs - launching Windows interop commands such as
cmd.exe /c echo cmd-okfrom WSL hangs - recovery requires
wsl --shutdownor restarting WSL services / rebooting
Relevant local checks
$ timeout 8 bash -lic 'printf shell-ok'
shell-ok
$ timeout 12 /mnt/c/Windows/System32/wsl.exe -d Ubuntu --exec /bin/bash -lc 'printf wsl-exec-ok'
<3>WSL (...) ERROR: UtilAcceptVsock:273: accept4 failed 110
$ timeout 10 /mnt/c/Windows/System32/wsl.exe --status
# timed out, no output
$ timeout 10 /mnt/c/Windows/System32/wsl.exe -l -v
# timed out, no output
$ timeout 8 /mnt/c/Windows/System32/cmd.exe /c echo cmd-ok
# timed out, no output
Journal excerpts
Repeated session leader failures:
Jun 12 10:04:35 ... WSL (849628 - SessionLeader) ERROR: UtilAcceptVsock:273: accept4 failed 110
Jun 12 10:04:35 ... WSL (849628 - SessionLeader) ERROR: operator():1263: UtilAcceptVsock() failed for session leader 110
Jun 12 10:04:44 ... WSL (849732 - SessionLeader) ERROR: UtilAcceptVsock:273: accept4 failed 110
Jun 12 10:04:44 ... WSL (849732 - SessionLeader) ERROR: operator():1263: UtilAcceptVsock() failed for session leader 110
Jun 12 15:47:51 ... WSL (1042443 - SessionLeader) ERROR: UtilAcceptVsock:273: accept4 failed 110
Jun 12 15:47:51 ... WSL (1042443 - SessionLeader) ERROR: operator():1263: UtilAcceptVsock() failed for session leader 110
Jun 12 17:10:40 ... WSL (1059662 - SessionLeader) ERROR: UtilAcceptVsock:273: accept4 failed 110
Jun 12 17:10:40 ... WSL (1059662 - SessionLeader) ERROR: operator():1263: UtilAcceptVsock() failed for session leader 110
Earlier memory / Hyper-V socket allocation failure:
Jun 11 17:40:07 ... kernel: kworker/0:0: page allocation failure: order:7, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
Jun 11 17:40:07 ... kernel: Workqueue: hv_pri_chan vmbus_add_channel_work
Jun 11 17:40:07 ... kernel: vmbus_alloc_ring+0x78/0xd0
Jun 11 17:40:07 ... kernel: vmbus_open+0x29/0x80
Jun 11 17:40:07 ... kernel: hvs_probe+0x1f4/0x4c0
Jun 11 17:40:07 ... kernel: vmbus_probe+0x42/0xa0
Jun 11 17:40:07 ... kernel: vmbus_device_register+0x87/0x130
Jun 11 17:40:07 ... kernel: vmbus_add_channel_work+0x146/0x1d0
Mem-Info from the same allocation failure showed heavy pressure near the WSL memory cap:
active_anon:114025 inactive_anon:11507801 isolated_anon:32
active_file:1031583 inactive_file:146747 isolated_file:0
free:60986 free_pcp:1372 free_cma:0
Node 0 active_anon:456100kB inactive_anon:46031204kB active_file:4126332kB inactive_file:586988kB
Node 0 Normal free:46896kB min:41688kB low:91140kB high:140592kB
Node 0 Normal: ... 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44724kB
Free swap = 12716576kB
Total swap = 13631488kB
The same allocation failure repeated at least two more times:
Jun 11 17:41:42 ... kernel: kworker/0:3: page allocation failure: order:7, mode:0xdc0(GFP_KERNEL|__GFP_ZERO)
Jun 11 17:41:42 ... kernel: Workqueue: hv_pri_chan vmbus_add_channel_work
Jun 11 17:41:42 ... kernel: vmbus_alloc_ring+0x78/0xd0
Jun 11 17:41:58 ... kernel: kworker/0:2: page allocation failure: order:7, mode:0xdc0(GFP_KERNEL|__GFP_ZERO)
Jun 11 17:41:58 ... kernel: Workqueue: hv_pri_chan vmbus_add_channel_work
Jun 11 17:41:58 ... kernel: vmbus_alloc_ring+0x78/0xd0
Notes
Docker Desktop is installed and WSL integration is active, but the installed Docker Desktop version is 4.44.3, newer than the 4.42.1 fix referenced from #12991 / docker/for-win#14656. This may still be Docker-triggered, but the local kernel evidence looks closer to #11612: vmbus_alloc_ring cannot allocate under memory pressure, then WSL session creation / interop starts timing out.
Activity
github-actions commentedon Jun 12, 2026
Logs are required for review from WSL team
If this a feature request, please reply with '/feature'. If this is a question, reply with '/question'.
Otherwise, please attach logs by following the instructions below, your issue will not be reviewed unless they are added. These logs will help us understand what is going on in your machine.
How to collect WSL logs
Download and execute collect-wsl-logs.ps1 in an administrative powershell prompt:
The script will output the path of the log file once done.
If this is a networking issue, please use
.\collect-wsl-logs.ps1 -LogProfile networkinginstead, following the instructions in Collect WSL logs for networking issuesOnce completed please upload the output files to this GitHub issue.
See Collect WSL logs (recommended method).
If you choose to email these logs instead of attaching them to the bug, please send them to wsl-gh-logs@microsoft.com with the GitHub issue number in the subject, and include a link to your GitHub issue comment in the message body, and reply with '/emailed-logs'.
kuza55 commentedon Jun 12, 2026
/emailed-logs
github-actions commentedon Jun 12, 2026
Diagnostic information
chemwolf6922 commentedon Jun 15, 2026
Hi kuza55 , I'm working on a fix for this. #40519.
If you would like to. Could you please try out this build: https://github.com/chemwolf6922/build-wsl/actions/runs/27520782738/artifacts/7628123738 to see if it resolves the issue? Thanks.
The build is at this commit: d41e13d
Please note this is a dev build. And may contain other issues. It's recommended to return to a released version after the test.
SeanIsTyping commentedon Jun 30, 2026
I encountered something similar running Podman and a PostgreSQL container. Already established connection continued running normally, and the guest Podman machine stays in the running state, but eventually no new connections to the database can be established, and most WSL commands will hang or return the Wsl/Service/0x80072747 error.
dmesg
Increasing the ram limit in
wslconfigto 8G does not help either. The same errors pops up after running the container for a day.exnavi commentedon Sep 14, 2026
Additional related observations from a Windows 10 / WSL host; seeking guidance before changing the system.
Environment: Windows 10 Home 22H2, build 19045.7725; WSL 2.7.12.0; kernel 6.18.33.2-microsoft-standard-WSL2; 7.87 GiB visible host RAM, 3.76 GiB guest RAM, 1 GiB guest swap.
Retained current-boot warnings show:
Later validated native Windows memory APIs measured 95% physical-memory load with about 384.5 MiB available and commit below its limit, near further allocation warnings. A later owner-local process snapshot showed vmmem as the largest working set (3.45 GiB); this was not synchronized with the failures. A Windows-backed directory scan also returned ENOMEM. An earlier passive interval was quiet and inconclusive. We have not established a relay leak or attributed the unavailable contiguous pages to a particular allocation owner.
Source comparison finds drivers/hv/channel.c identical between kernel tags linux-msft-wsl-6.18.33.2 and linux-msft-wsl-6.18.40.1, with p9_fcall_init and p9_client_readdir unchanged. This does not rule out indirect fixes elsewhere. WSL 2.7.14's process-accept call still lacks the explicit timeout added by PR #41252.
Is there an identified supported remedy for this combination, or a minimally scoped failure-time measurement that distinguishes ordinary capacity/fragmentation pressure from retained kernel/channel allocations? We are not assuming more swap, RAM or a stable update fixes it. No raw logs, private filenames or workload data are included here.
Prepared with coding-assistant assistance from retained local measurements; reviewed findings are limited as stated.
emersonbusson commentedon Sep 17, 2026
Technical Resolution & Reference Kernel Patches for Order-7 VMBus Ring Failures
Regarding the
vmbus_alloc_ringOrder-7 allocation failures andUtilAcceptVsock: accept4 failed 110timeouts under memory pressure:We encountered and systematically resolved this exact failure pattern on WSL2 (Kernel 6.18.40.1).
Root Cause:
vmbus_alloc_ring()requests contiguous physical pages viaalloc_pages(..., order: 7)(512 KiB). As developer workflows run, physical memory fragmentation depletes orders 7 through 10 in/proc/buddyinfo, even when several gigabytes of memory remain free across lower orders (orders 0–3). When the contiguous allocation fails, synthetic channel creation aborts, hanging the HCS session handshake.Tested Solution:
We implemented and validated a clean, two-part kernel resilience fix:
drivers/hv/ring_buffer.c,drivers/hv/channel.c):When
alloc_pages(order: 7)fails,vmbus_alloc_ringimmediately falls back tovzalloc(). Invmbus_establish_gpa_range(), the virtually mapped pages are translated to PFNs viavmalloc_to_page(). Because the Hyper-V host natively supports scattered PFNs in the GPA descriptor, this requires zero host-side changes and establishes channels in ≤ 0.15 ms under heavy fragmentation. Confidential VM (CoCo) memory isolation is strictly preserved.drivers/hv/hv_common.c,drivers/hv/hv_balloon.c):Guarantees 512 MiB dynamic atomic headroom and defers host balloon inflation (
si_mem_available() < totalram_pages() / 32) during active direct reclaim.Empirical Verification & Tested Tree:
Tested under sustained 99% RAM load with
PASS_ZERO_PANICand zero session timeouts. The full reference implementation, patches, and deployment instructions are available on:👉 emersonbusson/WSL2-Linux-Kernel (branches
linux-msft-wsl-6.18.yandfeature/ramshared-wsl2-resilience-6.18).chore(upstream): remove ephemeral submit helper and mark proposals as…
Hugo-Polloli commentedon Sep 22, 2026
I have observed this recurring on WSL 2.9.12.0 / kernel 6.18.40.1-microsoft-standard-WSL2, after upgrading from 2.6.3 and rebooting. This may help distinguish this allocation failure from the indefinite relay-accept lifetime fixed by #41252.
Environment: Windows 11 build 26200.9457; Ubuntu 22.04.5; 64 GB host RAM;
.wslconfigmemory=32GB, swap=8GB, networkingMode=mirrored. Docker Desktop integration and concurrent data-processing/development workloads are active. No synthetic memory stress test was run.The first order-7 failure I captured in
vmbus_alloc_ringoccurred about 41 minutes into the new boot. Later retained kernel output contains repeated order-7 ring allocation failures andUtilAcceptVsock: accept4 failed 110, plus order-4 Plan9p9_fcall_initfailures during ordinary Git reads. Existing Linux commands continue to work. The earlier first-failure excerpt was saved before the kernel ring buffer rotated.Evidence from a failure at uptime 9113 seconds:
A nearby, separately sampled
/proc/meminfoshowed approximately 26 GiB MemAvailable out of 31.35 GiB, dominated by buffers/file cache and reclaimable slab, with only about 250 MiB MemFree. I did not take this snapshot atomically with the failure. A retained order-4 failure also showed no blocks >=64 KiB in either zone. Cgroup accounting attributes most file-cache/metadata charge to two active data-processing services, with relatively small anonymous memory. I have not established which allocations prevent compaction or a specific triggering workload.In a later process-tree inspection, I found six relay processes, each still parented to a SessionLeader and with a child. A briefly observed unsuffixed Relay was young and had a live parent. This differs from the hours-old orphaned relays observed on the previous 2.6.3 boot; it does not exclude every possible channel/resource leak.
wsl-user/memory.eventshad zero max/oom/oom_kill events, and the guest OOM-kill counter was zero.compaction_proactivenessread as 0; it was not changed.In the exact running kernel tag,
vmbus_alloc_ringattempts a contiguous allocation on the target node and then viaalloc_pages, returning ENOMEM if both fail. The process-accept timeout fix therefore does not by itself remove this failure mode. This is an observed allocation mechanism, not a claim that I have proved the complete cause of fragmentation.I have since stopped the two data-processing services and the associated materializer, preserving their data. Their cached pages remain resident after exit; this is not yet a controlled demonstration of recovery or workload causation. I have not restarted them or run a cache-drop experiment.
Is there an official kernel fix planned for this case, or a preferred bounded diagnostic to identify why reclaim/compaction cannot satisfy these requests under cache-heavy workloads? I have not installed a custom kernel, dropped caches, changed VM tuning, or killed relay processes.
I prepared this with Codex assistance from local measurements and tagged source inspection. I have included only selected technical measurements and omitted raw private logs and workload identifiers.
ManthanVasani commentedon Sep 24, 2026
TL;DR: For connections the host opens into WSL init's listeners (session leader,
wsl.exestdio, interop), the order-7 ring is not inherent to hv_sock. It is sized from the listening socket's buffers, which WSL never sets, so it followsnet.core.wmem_default(212992: 52 + 52 pages, order 7). At 28672 the same connections need order 4. After that change, all 2,114 traced rings at the new size succeeded (the Plan 9 server, which sets its own 131072, still failed 5 of 51), and 294/294 interop calls succeeded, 84 of them with no free block of order 7 or larger. PR: #41690.Environment: WSL 2.7.3.0, kernel 6.6.114.1-microsoft-standard-WSL2, Windows 11 build 26200, networkingMode=mirrored, autoMemoryReclaim=gradual, 25 GB guest memory, 8 GB swap.
Where the order 7 comes from
hvs_open_connection()sizes both rings from the listener (skis the parent there):max(sk_sndbuf, 6 pages), capped at 64 pages (hyperv_transport.c#L372-L381).hvs_sock_init()(#L453-L454). That runs only atconnect()and for the accepted child (af_vsock.c#L1478, hyperv_transport.c#L342). It never runs for a stream socket'ssocket()orlisten()(af_vsock.c#L2406-L2412, #L1668-L1703).sock_init_data()copied fromwmem_default/rmem_default(sock.c#L3444-L3445). This dates from c0cfa2d8a788 ("vsock: add multi-transports support", v5.5). Before it, every socket ranhvs_sock_init().UtilListenVsockAnyPort()sets noSO_SNDBUF/SO_RCVBUF(util.cpp#L1567-L1618; same on master).vmbus_alloc_ring()allocates both rings as one physically contiguous block with no smaller fallback (channel.c#L167-L192): 104 pages, order 7. If that fails, the offer is dropped and the connection never reachesaccept(). That is what theaccept4 failed 110andWaiting for abnormally long acceptlines report.Measurement
Workaround, runtime only:
sudo sysctl -w net.core.wmem_default=28672 net.core.rmem_default=28672. I traced everyvmbus_alloc_ring()with a kprobe (commands below):-12)cmd.exe /c verpassed 294/294./proc/buddyinfo, read just before each run, showed no free block of order 7 or larger in DMA32 or Normal for 84 of them (64 KB+ blocks in those runs: 16 at the lowest, median about 480).hvs_probefailures in the ~5 h before the change (longest gap 90 min) and none in the ~5.2 h after. It is rate-limited, so the trace is the stronger evidence.Limits of the workaround
/etc/sysctl.d/misses the interop listener created at distro start.kernelCommandLine = sysctl.net.core.wmem_default=28672 sysctl.net.core.rmem_default=28672under[wsl2]should cover it, sincesysctl.*parameters apply before/init(main.c#L1478-L1481). I have not yet verified that after a restart.AF_UNIXdatagrams over ~28 KB fail withEMSGSIZE(af_unix.c#L1989). Bulk hv_sock paths also get small rings:wsl.exeand interop stdio, which init: size hvsocket listener buffers so host-initiated connections avoid order-7 ring allocations #41690 shrinks too (its throughput cost is still to be measured; see the PR),wsl --export/--import, and withkernelCommandLinethe NAT-mode localhost relay, which init: size hvsocket listener buffers so host-initiated connections avoid order-7 ring allocations #41690 leaves alone. TCP is unaffected.order:6warning fromhvs_probe, and each came milliseconds after 28672 rings in the same burst had succeeded.p9_fcall_init, readdir buffers) continued, at least 20 since the change, while every 28672 ring succeeded.Proposed fix
SO_SNDBUF/SO_RCVBUFto 14336 (28672 after doubling). The bulk listeners, WSLC's thread forks (which include thedocker.sockrelay) and Plan 9 keep their sizes. Guest-initiated connections need no change: they are already order 4, andhvs_sock_init()resets anySO_SNDBUFset beforeconnect(), soUtilConnectVsock'sSocketBufferdoes not size their rings.hvs_open_connection(), use the parent's sizes only if userspace set them (SOCK_SNDBUF_LOCK/SOCK_RCVBUF_LOCK), andRINGBUFFER_HVS_*_SIZEotherwise, which restores the pre-5.5 default. A non-contiguous ring ([Kernel Proposal] Prevent VMBus Control-Plane Starvation & High-Order Ring Allocation Deadlocks Under Memory Pressure #41634, New WSL sessions hang under memory pressure with vmbus_alloc_ring allocation failure #40795 (comment)) would remove the high-order requirement entirely.Reproducing the ring size (as root)
Any machine shows the ring size;
ret=-12needs fragmented memory. To undo the workaround, run the samesysctlwith 212992 (new sockets only).init: size hvsocket listener buffers so host-initiated connections av…
emersonbusson commentedon Sep 24, 2026
I am following up on this report and Michael Kelley's September 22 review of my
[PATCH 2/2] hv: vmbus: add virtual memory fallback for ring buffer allocations under memory pressureemail thread.I am preparing a kernel-side change that routes VMBus ring allocation through the existing
vmbus_alloc_buffer()mechanism, with chunked allocation and per-chunk Confidential VM handling. The direction matches the review's suggestion to use that mechanism for rings and keep buffer/GPADL lifetime state together. This is still a local draft; it has not been submitted, and fault-injection plus GPADL teardown and CoCo validation remain outstanding. I am not claiming that the issue has been reproduced or fixed by this draft.I also noticed the related WSL PR #41690, which reduces ring sizes for selected host-initiated hv_sock listeners. That is a useful, narrower mitigation for those WSL control and stdio paths. The kernel allocator change has a broader intended scope across VMBus ring users; the two approaches should be coordinated so the kernel series complements the WSL listener change rather than duplicates its coverage.
ManthanVasani commentedon Sep 24, 2026
EMERSON BUSSON (@emersonbusson) Thanks, agreed they complement each other. #41690 shrinks the rings of most of WSL's control and stdio listeners and needs no kernel change, so it helps on the 6.6 and 6.18 kernels WSL ships now. Allocating rings through
vmbus_alloc_buffer()would also cover what #41690 leaves alone: the Plan 9 server (order 6), and the bulk relays and WSLC thread forks (order 7).vmbus_alloc_buffer()is in mainline only since 7.3-rc1, so it needs a backport to reach WSL's kernels; after that, #41690 still cuts the control and stdio rings from 208 KB to 28 KB per direction (32 KB on 6.18).The hv_sock sizing idea from my comment above (listeners that set no
SO_SNDBUF/SO_RCVBUFhave keptwmem_default/rmem_defaultsince c0cfa2d8a788) is innet/vmw_vsock/hyperv_transport.c, so it would be a small hv_sock patch of its own rather than part of the VMBus series. It would also shrink WSL's bulk listeners to 24 KB rings (28 KB on 6.18), so WSL would need to size those explicitly to keep their throughput, and your change is what would let them keep large rings without needing contiguous memory.visa360-org commentedon Oct 15, 2026
Another data point, on the latest release: WSL 3.0.3.0, kernel 6.18.54.1-microsoft-standard-WSL2, Windows 10.0.26200.9711. Same trace as the report, and what we measured matches ManthanVasani's analysis above (the ring is sized from the listener's default socket buffers).
Setup:
.wslconfigsets onlymemory=5GBandswap=8GB(guest MemTotal 4919 MiB, 8 vCPUs, Ubuntu with systemd). Load: two or three parallel sessions running a Node/TypeScript monorepo build and its test suites (type-checking, unit tests, browser tests with Chromium).Frequency: 11 order-7 failures in about 15 hours, 2026-10-14 17:21 to 2026-10-15 08:43 (UTC+8; wall times converted from the kernel's monotonic clock), over three VM boots:
wsl --shutdownEach one was followed 30 s later by
UtilAcceptVsock ... accept4 failed 110from a Relay or SessionLeader, andwsl.exefailed withWsl/Service/WSAETIMEDOUT, while the sessions already open kept running. Two of them fell inside long hangs (about 100 and 70 minutes, both ended withwsl --shutdown) under heavy memory pressure: swap full in one, 5.8 of 8 GB used in the other. But at 18:46:45 the 8 GB swap was still unused, and 08:43:30 was fragmentation alone, with 1.8 GB of page cache, 2.5 GB anon, swap almost unused and no free block above 256 KiB:Ring size: with the default
net.core.wmem_default/rmem_defaultof 212992, the channels of each new session showedwrite_avail=208896(/sys/bus/vmbus/devices/*/channels/*/write_avail): two 208 KiB rings, one order-7 block. As expected while #41690 is still open.Mitigation applied (guest only, no restart):
/etc/sysctl.d/99-wsl-hvsock.conf, loaded withsysctl --system:First effect:
wsl.exesession after the change showwrite_avail=61440(64 KiB rings, an order-5 block). So do the four channels of an interop call (a Windows exe started from the guest), in a distro run that had started before the change. Channels opened earlier keep 208896.MemAvailable(4,264,168 kB to 3,842,500 kB at the moment of applying).Investigated and drafted by Claude Code (claude-opus-5-5) on behalf of the reporter.
0xdhx commentedon Oct 31, 2026
Adding a contributing cause for anyone here who also runs Docker Desktop: on the WSL2 backend it turns off both of the kernel's background anti-fragmentation mechanisms for the whole VM, which makes these order-7 failures much more likely.
Docker Desktop's init daemon in the
docker-desktopdistro (docker-desktop.iso:/initd, 4.97.0) writes a sysctl table into the shared kernel at every start, after the user distro'ssystemd-sysctl. It includes:So in every distro on the machine, proactive compaction is off and the watermark boost that normally reacts to fragmentation events never fires. On WSL 3.0.3.0 / kernel 6.18.54.1 with Docker running, all 222 order-7 failure reports in one boot show
Normal ... boost:0kB, and at failure the Normal zone often had hundreds of MiB (once 11 GB) free with no block of order 7 or larger. That is fragmentation, not a shortage of memory. One failure from this morning:Check with
sysctl vm.compaction_proactiveness vm.watermark_boost_factor;0 0while Docker Desktop is running is this. A/etc/sysctl.dfile alone does not hold, because Docker writes after it; re-applying it on a timer (systemd-sysctlevery few minutes) does, until the next Docker start. Reported to Docker as docker/desktop-feedback#809.This complements rather than replaces the listener-sizing fix in #41690: Docker's settings only remove the kernel's defences, while the order-7 request itself comes from the unsized listeners described above.