dask / dask/distributed

Logging configuration using environment variables

Open
#8,570 1 comment 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Describe the issue**:
The logging configuration can not be set using environment variables. However, it is the easiest way to pass some configuration to the cluster's workers. Specifically, I would like to update the following settings:
```python
dask.config.get("distributed.logging")
```
```
{'distributed': 'debug', 'distributed.client': 'debug', 'distributed.scheduler': 'debug', 'distributed.shuffle': 'debug'}
```

**Minimal Complete Verifiable Example**:
The following code throws an error
```python
import os

os.environ['DASK_DISTRIBUTED__LOGGING__DISTRIBUTED'] = 'info'
os.environ['DASK_DISTRIBUTED__LOGGING__DISTRIBUTED__CLIENT'] = 'info'
os.environ['DASK_DISTRIBUTED__LOGGING__DISTRIBUTED__SCHEDULER'] = 'info'
os.environ['DASK_DISTRIBUTED__LOGGING__DISTRIBUTED__SHUFFLE'] = 'info'

import dask
import dask.distributed
```

Trace:
```text
Traceback (most recent call last):
File ".../test.py", line 8, in
import dask.distributed
File ".../.venv/lib/python3.10/site-packages/dask/__init__.py", line 3, in
from dask import config, datasets
File ".../.venv/lib/python3.10/site-packages/dask/config.py", line 802, in
refresh()
File ".../.venv/lib/python3.10/site-packages/dask/config.py", line 522, in refresh
update(config, collect(**kwargs))
File ".../.venv/lib/python3.10/site-packages/dask/config.py", line 485, in collect
configs.append(collect_env(env=env))
File ".../.venv/lib/python3.10/site-packages/dask/config.py", line 270, in collect_env
set(d, config=result)
File ".../.venv/lib/python3.10/site-packages/dask/config.py", line 391, in __init__
self._assign(key.split("."), value, config)
File ".../.venv/lib/python3.10/site-packages/dask/config.py", line 458, in _assign
self._assign(keys[1:], value, d[key], path, record=record)
File ".../.venv/lib/python3.10/site-packages/dask/config.py", line 458, in _assign
self._assign(keys[1:], value, d[key], path, record=record)
File ".../.venv/lib/python3.10/site-packages/dask/config.py", line 458, in _assign
self._assign(keys[1:], value, d[key], path, record=record)
File ".../.venv/lib/python3.10/site-packages/dask/config.py", line 450, in _assign
d[key] = value
TypeError: 'str' object does not support item assignment
```

**Environment**:

- Dask version: 2024.1.0
- Python version: 3.10
- Operating System: Win11, WSL
- Install method (conda, pip, source): poetry

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.