dask / dask/distributed

Clarification on worker resources documentation

Open
#3,552 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

In the [worker resources docs](https://distributed.dask.org/en/latest/resources.html#resources-are-applied-separately-to-each-worker-process), it shows the following scenario.

> ```
> dask-worker scheduler:8786 --nprocs 3 --nthreads 2 --resources "process=1"
> ```
>
> With the code below, there will be at most 3 tasks running concurrently and each task will run in a separate process:
> ```
> from distributed import Client
> client = Client('scheduler:8786')
>
> futures = [client.submit(non_thread_safe_function, arg,
> resources={'process': 1}) for arg in args]
> ```
>

If additional tasks are scheduled without the `process` resource, might they be scheduled on the unused threads? The scenario I'm concerned about is when the `{process: 1}` consumes multiple threads, effectively starving these additional tasks.

If this is so, this leads me down the road of tagging all my additional tasks (and other workers) to avoid this situation. Consequently, I then lose out on scheduling on the `process` worker with these tasks when no `process` tasks are running. (Is this clear?)

I'd be happy to update the documentation to help elucidate this situation, once I feel I fully grok what is going on.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.