dask / dask/dask-kubernetes

Default DaskWorkerGroup is never created if daskcluster_create_components fails once

Open
#976 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
324
Forks
157
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:
A `DaskCluster` can end up `Running` without its `-default` `DaskWorkerGroup`. Its `DaskAutoscaler` then fails on every tick with:

```
kr8s._exceptions.NotFoundError: Could not find DaskWorkerGroup -default in namespace .
```

`daskcluster_create_components` is registered as `@kopf.on.field(..., field="status.phase", new="Created")`. If it fails part way through (for example a transient API error after the scheduler `Service` was created but before the worker group was), kopf schedules a retry. Meanwhile `handle_scheduler_service_status` fires for the new `Service` and patches `status.phase` to `Running`. The handler's `new="Created"` filter no longer matches, kopf stores the new diff base, and the pending retry is dropped. Nothing ever creates the default worker group after that.

Operator logs for an affected cluster look like:

```
Creating Dask cluster components.
Scheduler deployment -scheduler created in .
Scheduler service -scheduler created in .
Handler 'daskcluster_create_components/status.phase' failed with an exception and will try again in 60 seconds: ConnectError('')
```

with no `Worker group -default created` line afterwards and the cluster showing `phase: Running`.

Related: #968 is the same pair of handlers racing the other way (`Pending` overwriting `Running`).

**Anything else we need to know?**:

**Environment**:
- dask-kubernetes 2026.3.0

Contributor guide

Open the contributing guide

Research direction

Start by tracing the daskcluster_create_components and handle_scheduler_service_status handlers described in the issue, focusing on their status.phase filters and retry behavior. Reproduce or test a transient failure after the scheduler Service is created; done means the cluster reaches Running with its -default DaskWorkerGroup created, including coverage for the handler race.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.