Default DaskWorkerGroup is never created if daskcluster_create_components fails once
- Dominant language
- Python
- Stars
- 324
- Forks
- 157
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the issue**:
A `DaskCluster` can end up `Running` without its `-default` `DaskWorkerGroup`. Its `DaskAutoscaler` then fails on every tick with:
```
kr8s._exceptions.NotFoundError: Could not find DaskWorkerGroup -default in namespace .
```
`daskcluster_create_components` is registered as `@kopf.on.field(..., field="status.phase", new="Created")`. If it fails part way through (for example a transient API error after the scheduler `Service` was created but before the worker group was), kopf schedules a retry. Meanwhile `handle_scheduler_service_status` fires for the new `Service` and patches `status.phase` to `Running`. The handler's `new="Created"` filter no longer matches, kopf stores the new diff base, and the pending retry is dropped. Nothing ever creates the default worker group after that.
Operator logs for an affected cluster look like:
```
Creating Dask cluster components.
Scheduler deployment -scheduler created in .
Scheduler service -scheduler created in .
Handler 'daskcluster_create_components/status.phase' failed with an exception and will try again in 60 seconds: ConnectError('')
```
with no `Worker group -default created` line afterwards and the cluster showing `phase: Running`.
Related: #968 is the same pair of handlers racing the other way (`Pending` overwriting `Running`).
**Anything else we need to know?**:
**Environment**:
- dask-kubernetes 2026.3.0
Contributor guide
Research direction
Start by tracing the daskcluster_create_components and handle_scheduler_service_status handlers described in the issue, focusing on their status.phase filters and retry behavior. Reproduce or test a transient failure after the scheduler Service is created; done means the cluster reaches Running with its -default DaskWorkerGroup created, including coverage for the handler race.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100