dask / dask/distributed

LocalCluster does not apply worker resources as documented

Open
#7,261 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Describe the issue**:

`LocalCluster` objects don't seem to respect resources configured via environment variables, or via `dask.config`.

**Minimal Complete Verifiable Example**:

Explicitly specifying the available resources works correctly:

```python
from dask.distributed import Client, LocalCluster

cluster = LocalCluster(n_workers=1, resources={"GPU": 1})
c = Client(cluster)
c.cluster.workers[0].resources # {'GPU': 1}
```

However, [the documentation](https://distributed.dask.org/en/stable/resources.html#specifying-resources) suggests that specifying this via the environment variable `DASK_DISTRIBUTED__WORKER__RESOURCES__GPU` should also work, but it does not seem to (previously, I'd run `export DASK_DISTRIBUTED__WORKER__RESOURCES__GPU=1`):

```python3
os.environ["DASK_DISTRIBUTED__WORKER__RESOURCES__GPU"] # '1'

cluster = LocalCluster(n_workers=1)
c = Client(cluster)
c.cluster.workers[0].resources # None
```

The same documentation page also suggests a snippet with `dask.config.set`, I was unable to get this to produce a different result.

**Environment**:

- Dask version: `dask, version 2022.10.0`
- Python version: `Python 3.9.13`
- Operating System: `ubuntu:20.04` (Docker image)
- Install method (conda, pip, source): `pip`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.