dask / dask/dask-kubernetes

Current namespace not used when creating daskcluster in k8s

Open
#921 9 comments 1 reaction 0 assignees View on GitHub
needs info
Dominant language
Python
Stars
324
Forks
157
Avg merge
14d 17h
Merged PRs (30d)
1

Description

**Describe the issue**:

When using a dask operator deployment in k8s with the role/rolebinding defined at the namespace level (`rbac.cluster: false`), the creation of a `daskclusters.kubernetes.dask.org` by a service account (`dask` in the example) inside a pod within a namespace (`myns` in the example) leads to the following error:

**Short Error Message:**
```bash
User "system:serviceaccount:myns:dask" cannot create resource "daskclusters" in API group "kubernetes.dask.org" in the namespace "default"
...
User "system:serviceaccount:myns:dask" cannot list resource "daskclusters" in API group "kubernetes.dask.org" in the namespace "default"
```

Full Stacktrace:
```bash
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/kr8s/_api.py", line 168, in call_api
response.raise_for_status()
File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 829, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://.../apis/kubernetes.dask.org/v1/namespaces/default/daskclusters'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/prefect/engine.py", line 42, in
run_flow(flow, flow_run=flow_run)
File "/usr/local/lib/python3.11/site-packages/prefect/flow_engine.py", line 1453, in run_flow
return run_flow_sync(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/flow_engine.py", line 1333, in run_flow_sync
return engine.state if return_type == "state" else engine.result()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/flow_engine.py", line 313, in result
raise self._raised
File "/usr/local/lib/python3.11/site-packages/prefect/flow_engine.py", line 721, in run_context
yield self
File "/usr/local/lib/python3.11/site-packages/prefect/flow_engine.py", line 1331, in run_flow_sync
engine.call_flow_fn()
File "/usr/local/lib/python3.11/site-packages/prefect/flow_engine.py", line 744, in call_flow_fn
result = call_with_parameters(self.flow.fn, self.parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prefect/utilities/callables.py", line 206, in call_with_parameters
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/workdir/examples/cs/flows/misc/run_on_dask/flow.py", line 43, in run_on_dask
cluster = KubeCluster(
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 282, in __init__
self.sync(self._start)
File "/usr/local/lib/python3.11/site-packages/distributed/utils.py", line 363, in sync
return sync(
^^^^^
File "/usr/local/lib/python3.11/site-packages/distributed/utils.py", line 439, in sync
raise error
File "/usr/local/lib/python3.11/site-packages/distributed/utils.py", line 413, in f
result = yield future
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 766, in run
value = future.result()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 322, in _start
await self._create_cluster()
File "/usr/local/lib/python3.11/site-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 361, in _create_cluster
await cluster.create()
File "/usr/local/lib/python3.11/site-packages/kr8s/_objects.py", line 320, in create
async with self.api.call_api(
File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kr8s/_api.py", line 186, in call_api
raise ServerError(
kr8s._exceptions.ServerError: daskclusters.kubernetes.dask.org is forbidden: User "system:serviceaccount:myns:dask" cannot create resource "daskclusters" in API group "kubernetes.dask.org" in the namespace "default"
Exception ignored in atexit callback:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 1033, in reap_clusters
asyncio.run(_reap_clusters())
File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 1031, in _reap_clusters
cluster.close(timeout=10)
File "/usr/local/lib/python3.11/site-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 700, in close
return self.sync(self._close, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/distributed/utils.py", line 363, in sync
return sync(
^^^^^
File "/usr/local/lib/python3.11/site-packages/distributed/utils.py", line 439, in sync
raise error
File "/usr/local/lib/python3.11/site-packages/distributed/utils.py", line 413, in f
result = yield future
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/tornado/gen.py", line 766, in run
value = future.result()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/dask_kubernetes/operator/kubecluster/kubecluster.py", line 706, in _close
cluster = await DaskCluster.get(self.name, namespace=self.namespace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kr8s/_objects.py", line 265, in get
raise e
File "/usr/local/lib/python3.11/site-packages/kr8s/_objects.py", line 255, in get
resources = await api.async_get(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kr8s/_api.py", line 460, in async_get
async with self.async_get_kind(
File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kr8s/_api.py", line 396, in async_get_kind
async with self.call_api(
File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/kr8s/_api.py", line 186, in call_api
raise ServerError(
kr8s._exceptions.ServerError: daskclusters.kubernetes.dask.org "test-cluster" is forbidden: User "system:serviceaccount:myns:dask" cannot list resource "daskclusters" in API group "kubernetes.dask.org" in the namespace "default"
```

**Minimal Complete Verifiable Example**:

Running this inside a pod:
```python
from dask_kubernetes.operator.kubecluster.kubecluster import KubeCluster, make_cluster_spec

if __name__ == '__main__':
spec = make_cluster_spec(
name="test-cluster",
)
cluster = KubeCluster(
custom_cluster_spec=spec,
)

cluster.adapt(minimum=0, maximum=2)
```

**Anything else we need to know?**:

When running the exact same test with `2024.5.0` version, it works fine so I think this is due to an update made in the `2024.8.0` release since it does not work since this version.

To make this work with `2024.8.0` or later, I need to define the `namespace` option when instantiating the `KubeCluster` (but I don't know the ns in advance in my use case):

**Environment**:

- Dask version: 2024.11.2
- Python version: 3.11.9
- Operating System: ubuntu 22.04
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Research direction

Start in dask_kubernetes/operator/kubecluster/kubecluster.py, following KubeCluster and the _create_cluster and _close paths shown in the traceback. Reproduce the minimal example inside a pod using the service account in myns, then compare behavior with 2024.5.0; done means the current namespace is used without passing namespace explicitly and the create and cleanup operations succeed.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.