dask / dask/distributed

Workers not using environment variables for default config options

Open
#7,070 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**WARNING**: The behavior reported here may **not** be a bug. For example, the consensus may be: "You must set your environment variables before importing dask." However, I want to document the recent change in behavior, because it's breaking dask-cuda CI.

**What happened**:

The workers within a `LocalCluster` do not use environment variables to set default config options unless the environment variables were set **before** `dask` was imported.

**What you expected to happen**:

Environment variables should still be used to set `dask.config` defaults after importing `dask`.

**Minimal Complete Verifiable Example**:

```python
import os
import dask
from dask.distributed import LocalCluster, Client

def check_config():
assert os.environ.get("DASK_DISTRIBUTED__COMM__COMPRESSION") == "False"
print(f"distributed.comm.compression={dask.config.get('distributed.comm.compression')}")

os.environ["DASK_DISTRIBUTED__COMM__COMPRESSION"] = "False"
with LocalCluster(n_workers=1) as cluster:
with Client(cluster) as client:
client.run(check_config)

# Prints: distributed.comm.compression=auto
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.