dask / dask/distributed

rootish tasks with resources restrictions are not queued

Open
#9,164 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Hi.

I am using client.submit with resources restrictions for root tasks (only CPU only, not GPU)
Then these root tasks results (futures) are set as input to subsequent client.submit calls (which are then non root tasks) that need to run on GPU only.

In the scheduler.py, the function is_rootish returns False whenever a task has resources, or worker restrictions.

I don't understand why. This cause queuing to be INOP in my case, and memory gradually increasing up to cluster saturation because root tasks (CPU bound tasks) are fulfilled faster than the actual results consumption (non root GPU bound), basically because I have much more CPU workers that GPU workers.

Is this intended, and if yes why ?
Thanks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.