Can´t create Dask Cluster on GKE using Artifact Registry
- Dominant language
- Python
- Stars
- 148
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
**What happened**:
I am deploying Dask Gateway on a [GKE](https://cloud.google.com/kubernetes-engine) cluster using the helm chart.
I have setup a custom image that is stored in [Artifact Registry](https://cloud.google.com/artifact-registry) for the Schedulers and workers. I have used for that purpose the key `gateway.backend.image.name` of `values.yam` with an example value `europe-west6-docker.pkg.dev/project1/images/dask`.
When I try to create a cluster with:
```
from dask_gateway import Gateway
# -- Here we provide a few examples of creating a `Gateway` object --
# Gateway server running at http://mygateway.com with kerberos authentication
gateway = Gateway(
address="http://localhost:8000/"
)
options = gateway.cluster_options()
options
cluster = gateway.new_cluster(options)
cluster.scale(1)
```
Please, take in mind that I am using `localhost` as I made a port-forwad to the pod `traefik-dask-gateway`.
The code throws me the next error:
```
---------------------------------------------------------------------------
GatewayClusterError Traceback (most recent call last)
in
----> 1 cluster = gateway.new_cluster(options)
2 cluster.scale(1)
~/miniconda3/envs/Dask/lib/python3.7/site-packages/dask_gateway/client.py in new_cluster(self, cluster_options, shutdown_on_close, **kwargs)
641 cluster_options=cluster_options,
642 shutdown_on_close=shutdown_on_close,
--> 643 **kwargs,
644 )
645
~/miniconda3/envs/Dask/lib/python3.7/site-packages/dask_gateway/client.py in __init__(self, address, proxy_address, public_address, auth, cluster_options, shutdown_on_close, asynchronous, loop, **kwargs)
816 shutdown_on_close=shutdown_on_close,
817 asynchronous=asynchronous,
--> 818 loop=loop,
819 )
820
~/miniconda3/envs/Dask/lib/python3.7/site-packages/dask_gateway/client.py in _init_internal(self, address, proxy_address, public_address, auth, cluster_options, cluster_kwargs, shutdown_on_close, asynchronous, loop, name)
912 self.status = "starting"
913 if not self.asynchronous:
--> 914 self.gateway.sync(self._start_internal)
915
916 @property
~/miniconda3/envs/Dask/lib/python3.7/site-packages/dask_gateway/client.py in sync(self, func, *args, **kwargs)
337 )
338 try:
--> 339 return future.result()
340 except BaseException:
341 future.cancel()
~/miniconda3/envs/Dask/lib/python3.7/concurrent/futures/_base.py in result(self, timeout)
430 raise CancelledError()
431 elif self._state == FINISHED:
--> 432 return self.__get_result()
433 else:
434 raise TimeoutError()
~/miniconda3/envs/Dask/lib/python3.7/concurrent/futures/_base.py in __get_result(self)
382 def __get_result(self):
383 if self._exception:
--> 384 raise self._exception
385 else:
386 return self._result
~/miniconda3/envs/Dask/lib/python3.7/site-packages/dask_gateway/client.py in _start_internal(self)
926 self._start_task = asyncio.ensure_future(self._start_async())
927 try:
--> 928 await self._start_task
929 except BaseException:
930 # On exception, cleanup
~/miniconda3/envs/Dask/lib/python3.7/site-packages/dask_gateway/client.py in _start_async(self)
944 # Connect to cluster
945 try:
--> 946 report = await self.gateway._wait_for_start(self.name)
947 except GatewayClusterError:
948 raise
~/miniconda3/envs/Dask/lib/python3.7/site-packages/dask_gateway/client.py in _wait_for_start(self, cluster_name)
576 raise GatewayClusterError(
577 "Cluster %r failed to start, see logs for "
--> 578 "more information" % cluster_name
579 )
580 elif report.status is ClusterStatus.STOPPED:
GatewayClusterError: Cluster 'dask-gateway.31f49455387e410ead07c154aa126e1a' failed to start, see logs for more information
```
I can´t find any logs with a more detailed error.
The fact is that If I store that image in other Docker image repositories it is working.
What Kubernetes Service Account is using Dask Gateway for pulling the images? I don´t see any pod getting launched even with a `PullImageError` or a similar error.
**What you expected to happen**:
I would expect the cluster to be created like with images stored in other repositories.
```python
# Put your MCVE code here
```
**Anything else we need to know?**:
**Environment**:
- Dask version:
- Python version:
- Operating System:
- Install method (conda, pip, source):
Contributor guide
Assessment
This issue has not been assessed yet.