dask / dask/dask-kubernetes

Handle `image`, `env` and `args` fields updates in DaskCluster in k8s operator

Open
#895 6 comments 0 reactions 0 assignees View on GitHub
enhancement operator
Dominant language
Python
Stars
324
Forks
157
PR merge metrics
No merged PRs in 30d

Description

I have a permanent dask cluster in kubernetes. Current operator ignores all changes to manifest.
There has been an issue about supporting spec updates, it got closed as resolved after implementing scale field support: https://github.com/dask/dask-kubernetes/issues/636.

The only fields that cause changes to deployment after applying updated manifest are `spec.worker.replicas`, `DaskAutoscaler` min/max.

Is it possible to support other fields, specifically `image`, `args`, `env`, `volumes/mounts`?
If not, what could be the optimal way to gracefully shut down and update cluster?

Cluster manifest (mostly copypasted from example):
```yaml
---
apiVersion: kubernetes.dask.org/v1
kind: DaskCluster
metadata:
name: dask-primary
spec:
worker:
replicas: 1
spec:
containers:
- name: worker
image: "//backend/dask:image"
imagePullPolicy: Always
args:
- worker
- --name
- $(DASK_WORKER_NAME)
- --dashboard
- --dashboard-address
- "8788"
ports:
- name: http-dashboard
containerPort: 8788
protocol: TCP
env:
- name: ENV_1
value: 1
- name: ENV_2
value: 2
volumeMounts:
- name: kafka-certs
mountPath: /etc/ssl/kafka/ca.crt
subPath: ca.crt
readOnly: true

volumes:
- name: kafka-certs
configMap:
name: kafka-certs

scheduler:
spec:
containers:
- name: scheduler
image: "//backend/dask:image"
imagePullPolicy: Always
args:
- scheduler
ports:
- name: tcp-comm
containerPort: 8786
protocol: TCP
- name: http-dashboard
containerPort: 8787
protocol: TCP
readinessProbe:
httpGet:
port: http-dashboard
path: /health
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
port: http-dashboard
path: /health
initialDelaySeconds: 15
periodSeconds: 20
imagePullSecrets:
- name: regcred
service:
type: ClusterIP
selector:
dask.org/cluster-name: dask-primary
dask.org/component: scheduler
ports:
- name: tcp-comm
protocol: TCP
port: 8786
targetPort: "tcp-comm"
- name: http-dashboard
protocol: TCP
port: 8787
targetPort: "http-dashboard"

---
apiVersion: kubernetes.dask.org/v1
kind: DaskAutoscaler
metadata:
name: dask-primary
spec:
cluster: dask-primary
minimum: 1
maximum: 10
```

Operator version: `helm install --repo https://helm.dask.org --create-namespace -n dask-operator --generate-name --version 2024.5.0 dask-kubernetes-operator`
Dask version: custom built image that uses the following deps:
```
dask = "^2024.5.2"
bokeh = "^3.4.1"
distributed = "^2024.5.2"
```
Although it's the same with `2024.5.2-py3.11` image

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the manifest updates with operator version 2024.5.0 and read issue #636 alongside the DaskCluster example. Trace how changes to worker and scheduler image, args, env, volumes, and mounts are handled; done means supported updates are applied safely, or the graceful shutdown and update path is clearly established.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.