rootish tasks with resources restrictions are not queued #9164

Activity

1. github-actions

   added

   needs triage

   on Dec 12, 2025
2. vianneylanguille

   changed the title [-]rootish tasks with resources restrictions[/-] [+]rootish tasks with resources restrictions are not queued[/+] on Dec 12, 2025
3. jacobtomlinson

   added

   needs attentionIt's been a while since this was pushed on. Needs attention from the owner or a maintainer.It's been a while since this was pushed on. Needs attention from the owner or a maintainer.

   and removed

   needs triage

   on Dec 15, 2025
4. crusaderky commented on Jan 30, 2026

   

   crusaderky

   on Jan 30, 2026

   Collaborator

   More actions

   This is a long-standing issue: #8177
5. shivmodi commented on May 9, 2026

   

   shivmodi

   on May 9, 2026

   More actions

   Hi.

   I am using client.submit with resources restrictions for root tasks (only CPU only, not GPU) Then these root tasks results (futures) are set as input to subsequent client.submit calls (which are then non root tasks) that need to run on GPU only.

   In the scheduler.py, the function is_rootish returns False whenever a task has resources, or worker restrictions.

   I don't understand why. This cause queuing to be INOP in my case, and memory gradually increasing up to cluster saturation because root tasks (CPU bound tasks) are fulfilled faster than the actual results consumption (non root GPU bound), basically because I have much more CPU workers that GPU workers.

   Is this intended, and if yes why ? Thanks

   Hi @vianneylanguille and maintainers,

   I’m interested in helping with this issue. My plan is to investigate why rootish tasks with resource restrictions are not queued and propose a fix in scheduler.py.

   Before I start, I wanted to check:

   • Are there any design considerations I should be aware of regarding root tasks with CPU/GPU constraints?
   • Is there an existing preferred approach for handling resource-based queuing that I should follow?

   I’d love to get feedback before I submit a PR. Thanks!
6. vianneylan commented on May 11, 2026

   

   vianneylan

   on May 11, 2026

   More actions

   Hello !

   Thanks for diving into it !

   I am not very familiar with Dask's internals, but at least I can tell a bit more about my use case:

   • the tasks I have can be executed either on GPU or on CPU. This seems obvious to me but apparently few people are concerned of it. Thus if a GPU is available it should preferably run on it, but it there is not, we don't want it to wait for GPU. Strict resources requirements prevent this, because they are strict. I attempted annotations which are reported to be soft resources requirements, but it did not work.
   • anyway regardless of the above point, there is the queuing issue you are mentioning.
   • in general, python code is often written to be either GPU or CPU compatible with very similar code base. eg Cupy or Torch which mimic a large part of Numpy API. Thus the code needs to know whether it is being run on a GPU worker. The only way I found to achieve it was to set the name of the GPU worker (which is a dask-cuda worker) like "GPU-xx" and parse the worker name inside the task to find "GPU" word in it, because the "resources" dictionary of worker executing the task is not accessible, for some reason.

   I ended up with dropping completely Dask resources management, and using a simple Semaphore based GPU management process enabled my first bullet point above to work fine. This solved also the queuing issue as no resources anymore.

   Vianney
7. crusaderky commented on May 15, 2026

   

   crusaderky

   on May 15, 2026

   Collaborator

   More actions

   I’m interested in helping with this issue. My plan is to investigate why rootish tasks with resource restrictions are not queued and propose a fix in scheduler.py.

   Before I start, I wanted to check:

   • Are there any design considerations I should be aware of regarding root tasks with CPU/GPU constraints?
   • Is there an existing preferred approach for handling resource-based queuing that I should follow?

   I’d love to get feedback before I submit a PR. Thanks!

   I would not advise someone who is new to dask development to start from this issue. There is good reason why this feature was left out when scheduler-side queuing was introduced a few years ago: it is quite nuanced and getting it wrong can easily cause either performance degradation for everyone (including users that don't use resources), deadlocks, or a crashed scheduler.

   Please be aware that maintainers will not review PRs written by insufficiently supervised AI - e.g. whenever the developer would be incapable of solving the problem on their own.
8. crusaderky

   removed

   needs attentionIt's been a while since this was pushed on. Needs attention from the owner or a maintainer.It's been a while since this was pushed on. Needs attention from the owner or a maintainer.

   on May 28, 2026