Skip to content

numba threadpool under dask results in thread oversubscription #1798

Closed

Description

@cmdupuis3

When a numba kernel with parallel=True is called from a dask worker thread, numba starts its own thread pool inside dask's. What happens next depends on numba's threading layer. Under workqueue the whole process aborts ("Concurrent access has been detected"); running a percentile over a chunked neighborhood reduction on dask's threaded scheduler reproduces it today. Under omp it doesn't crash, but every calling thread gets a full thread team, so a 12-thread pool grows to 179 OS threads. Only tbb handles it cleanly, and users get whichever layer happens to be installed.

Activity

  1. self-assigned this
    on Sep 29, 2026
  2. changed the title [-]numba threadpool under dask results in a thread bomb[/-] [+]numba threadpool under dask results in thread oversubscription[/+] on Sep 29, 2026
  3. moved this from 📚 Backlog to 📝 To-Do in UXarray Developmenton Oct 5, 2026
  4. moved this from 📝 To-Do to 🏗 In progress in UXarray Developmenton Oct 19, 2026
  5. moved this from 🏗 In progress to 👀 In review in UXarray Developmenton Oct 21, 2026
  6. moved this from 👀 In review to ✅ Done in UXarray Developmenton Oct 23, 2026
  7. added a commit that references this issue on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

scalabilityRelated to scalability & performance efforts

Type

No type

Fields

Priority

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions