dask / dask/dask-kubernetes

Controller loses API connection after token expiry on Azure Kubernetes Service (AKS)

Open
#964 3 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
324
Forks
157
PR merge metrics
No merged PRs in 30d

Description

dask-operator sometimes fails to create Dask Jobs on Azure Kubernetes Service (AKS) 1.33.5:

**Related issues**: https://github.com/dask/dask-kubernetes/issues/913

It looks like issue is not completely fixed. We have seen two occurrences of this issues in last two weeks.

Also, see https://github.com/nolar/kopf/issues/980#issuecomment-2436497975 report for details.

Logs:
```
[2025-12-01 22:20:46,800] kr8s._auth [DEBUG ] Reloading credentials
...
[2025-12-01 22:20:46,822] httpcore.http11 [DEBUG ] send_request_headers.started request=
[2025-12-01 22:20:46,822] httpcore.http11 [DEBUG ] send_request_headers.complete
...
[2025-12-01 22:20:46,901] httpx [INFO ] HTTP Request: GET https://10.0.0.1/apis/kubernetes.dask.org/v1/namespaces/join/daskclusters/job-d90c9c3c-0ae2-48aa-ba00-c80da3bce657 "HTTP/1.1 401 Unauthorized"
```

Minimal Complete Verifiable Example:

Install dask-operator on AKS 1.33.5
Wait an hour for the authentication token to expire.
Create a DaskJob resource.

dask-operator will not create the DaskJob because dask-operator's kubernetes authentication token has expired and kopf's watchers are no longer connected to kubeapi. A bug in kopf prevents kopf from refreshing the authentication token.

This only occurs on AKS 1.30+ because that is AKS now sets --service-account-extend-token-expiration to false.

Environment:
Dask operator version: 2025.7.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.