dask / dask/dask-kubernetes

Cannot Overwrite DASK_SCHEDULER_ADDRESS in Worker env

Open
#873 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
324
Forks
157
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:
To use a different port or non default FQDN for the scheduler we provide `DASK_SCHEDULER_ADDRESS` env in `daskcluster` `.spec.worker.spec.containers[0].env`, however the operator creates 2 items in worker deployment env. The last one overrides the first in the pod's env. So we cannot change `DASK_SCHEDULER_ADDRESS` environment variable from dask kubernetes objects.

**Minimal Complete Verifiable Example**:

```python
from dask_kubernetes.operator import KubeCluster
cluster = KubeCluster(name="test", n_workers=1, namespace="dask-operator", env={"DASK_SCHEDULER_ADDRESS": "tcp://test-scheduler:8786"})
```
DaskWorkerGroup
```yaml
env:
- name: DASK_SCHEDULER_ADDRESS
value: tcp://dev-cluster-scheduler:8786
```

Worker Pod
```yaml
env:
- name: DASK_SCHEDULER_ADDRESS
value: tcp://dev-cluster-scheduler:8786
- name: DASK_WORKER_NAME
value: dev-cluster-default-worker-5f4e5766de
- name: DASK_SCHEDULER_ADDRESS
value: tcp://dev-cluster-scheduler.dask-operator.svc.cluster.local:8786
```

**Anything else we need to know?**:

Duplicate items with same names in container["env"] and env should be removed from env before adding it to configuration
Current:
https://github.com/dask/dask-kubernetes/blob/b668cc62944dc3eac225dc4c4ff01f2b3743c9da/dask_kubernetes/operator/controller/controller.py#L166-L170
https://github.com/dask/dask-kubernetes/blob/b668cc62944dc3eac225dc4c4ff01f2b3743c9da/dask_kubernetes/operator/controller/controller.py#L202-L206

Fix:
```py
for container in deployment_spec["spec"]["template"]["spec"]["containers"]:
if "env" in container:
# Remove duplicate env vars
container_env_names = [env_item["name"] for env_item in container["env"]]
for env_item in env:
if env_item["name"] in container_env_names:
env.remove(env_item)
# Add the env vars
container["env"].extend(env)
else:
container["env"] = env
```
**Environment**:
2024.3.0 dask operator

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.