Repository navigation
CI Intermittent Windows fatal exception: access violation in tests/empirical_scope_observation.py #261
Description
Partly generated by an agent, but I did double-check with my brain on 😉.
cc @itamarst in case you have some insights or you have seen something similar when putting this empirical tests together.
I may try to investigate further on my Windows VM to see if I can at least reproduce.
The problem
The Windows pylatest_conda_forge_openblas CI job intermittently crashes while running:
PYTHONPATH=. python -X faulthandler tests/empirical_scope_observation.py blas
+ python -X faulthandler tests/empirical_scope_observation.py blas
Windows fatal exception: access violation
Thread 0x00000f48 [ThreadPoolExecutor-0_0Windows fatal exception: ]access violation
(most recent call first):
Current thread's C stack trace (most recent call first):
<cannot get C stack on this system>
File "D:\a\threadpoolctl\threadpoolctl\tests\empirical_scope_observation.py", line 58 in blas_math
File "C:\Users\runneradmin\miniconda3\envs\testenv\Lib\concurrent\futures\thread.py", line 73 in run
File "C:\Users\runneradmin\miniconda3\envs\testenv\Lib\concurrent\futures\thread.py", line 86 in run
File "C:\Users\runneradmin\miniconda3\envs\testenv\Lib\concurrent\futures\thread.py", line 119 in _worker
File "C:\Users\runneradmin\miniconda3\envs\testenv\Lib\threading.py", line 1024 in run
Windows fatal exception: access violation
Current thread's C stack trace (most recent call first):
<cannot get C stack on this system>
./continuous_integration/run_tests.sh: line 34: 1491 Segmentation fault PYTHONPATH=. python -X faulthandler tests/empirical_scope_observation.py blas
Error: Process completed with exit code 139.
Unfortunately, faulthandler doesn't show anything useful on Windows hence the cannot get C stack on this system
Environment
The September 24 failure and September 26 successful run both use:
- Python 3.14.7, with the GIL enabled
- NumPy 2.5.3
- conda-forge OpenBLAS 0.3.34, pthreads build
- The same threadpoolctl commit,
f6bef5dca966327941a87f4a92673cd9c0096add
CI logs
It seems to happen quite often
- 💔 September 22
- 💔 September 24
- 💔 September 25
- ✔️ September 26
- 💔 September 27
To investigate further
It could be useful to capture a native crash dump with ProcDump and inspect it with WinDbg.
LLM suggestion
For this CI crash, use ProcDump to capture a dump, then inspect it with WinDbg. After downloading Microsoft ProcDump, run this in PowerShell with the conda environment activated and the repository as the working directory:
$env:PYTHONPATH = "."
New-Item -ItemType Directory -Force dumps | Out-Null
.\procdump64.exe -accepteula -ma -e 1 -f C0000005 -n 1 `
-x dumps "$env:CONDA_PREFIX\python.exe" `
-X faulthandler tests/empirical_scope_observation.py blas
This captures a full dump at the first access violation (C0000005), before application exception handlers run. Upload dumps/*.dmp as a CI artifact, using if: always() on the upload step. ProcDump options
Open the dump in WinDbg and run:
.symfix
.reload
!analyze -v
.ecxr
kv
~* kb
.ecxr selects the exception context; kv shows its native stack; ~* kb shows every thread’s stack.
Microsoft debugging guidance
Symbol availability determines the detail: without matching symbols, you may get DLL names and offsets rather than internal function names or source lines. Preserve the exact Python, NumPy and OpenBLAS binaries from the failing environment, plus any matching debug symbols. Even an unsymbolized trace can help establish which native library is crashing.
Activity
[-]Intermittent segfault on Windows in tests/empirical_scope_observation.py[/-][+]CI Intermittent segfault on Windows in `tests/empirical_scope_observation.py`[/+][-]CI Intermittent segfault on Windows in `tests/empirical_scope_observation.py`[/-][+]CI Intermittent access violation on Windows in `tests/empirical_scope_observation.py`[/+][-]CI Intermittent access violation on Windows in `tests/empirical_scope_observation.py`[/-][+]CI Intermittent `Windows fatal exception: access violation` in `tests/empirical_scope_observation.py`[/+]lesteve commentedon Oct 1, 2026
I am unable to reproduce the access violation on my Windows VM so I'll try to debug in the CI, see #262
itamarst commentedon Oct 1, 2026
Might be this? OpenMathLib/OpenBLAS#5954
Not fixed in 0.3.34, so if they did a release that would get it out there. And then get scipy et all to rebuild their wheels.
There's a bunch of thread-safety issues in OpenBLAS that were fixed recently, and plausibly there might be more.
lesteve commentedon Oct 2, 2026
Not 100% sure, but the summary of my investigation seems to point to OpenBLAS CPU-specific issues but they also have been fixed in the
developbranch. Since the failure rate is ~20% (not too high), I would say, let's wait until next OpenBLAS release and hope it goes away 🤞?itamarst commentedon Oct 28, 2026
I asked OpenBLAS for a release so we can tell if it's already fixed or a new, still unfixed issue.
lesteve commentedon Oct 29, 2026
Nice thanks for asking OpenBLAS! I am sure an agent is enough to help you get a working setup on Windows, but feel free to ask if you get stuck or share if you end up with a setup you like 🙏.
Personnally for Windows testing this is what I do:
I am also investigating whether we can have a shared Windows VM on the Probabl side, but this would be Probabl-specific and that wouldn't be very useful for you 😅.