dask / dask/dask-gateway

usage of `dask.distributed.Queue` raises ValueError: unknown address scheme 'gateway'

Open
#404 9 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
148
Forks
93
PR merge metrics
No merged PRs in 30d

Description

**What happened**:

When i try to use `dask.distributed.Queue` on workers of a `dask-gateway` cluster i get `ValueError: unknown address scheme 'gateway'`

**What you expected to happen**:

I expect `dask.distributed.Queue` to work, as it does e.g. in a `dask.distributed.LocalCluster`

**Minimal Complete Verifiable Example**:

```sh
python -m venv ~/venv/dask-gateway
source ~/venv/dask-gateway/bin/activate
pip install dask-gateway 'dask-gateway-server[local]'

dask-gateway-server
```

```python
from dask_gateway import Gateway
from dask.distributed import Queue
gateway = Gateway("http://127.0.0.1:8000")
cluster = gateway.new_cluster()
cluster.scale(1)
client = cluster.get_client()
q = Queue("log")

def log_something():
q.put(42)

client.submit(log_something).result()
```

results in

```
Traceback (most recent call last):
File "", line 1, in
File "/home/nikolai/venv/dask-gateway/lib/python3.9/site-packages/distributed/client.py", line 227, in result
raise exc.with_traceback(tb)
File "/home/nikolai/venv/dask-gateway/lib/python3.9/site-packages/distributed/protocol/pickle.py", line 75, in loads
return pickle.loads(x)
File "/home/nikolai/venv/dask-gateway/lib/python3.9/site-packages/distributed/queues.py", line 289, in __setstate__
client = get_client(address)
File "/home/nikolai/venv/dask-gateway/lib/python3.9/site-packages/distributed/worker.py", line 3615, in get_client
address = comm.resolve_address(address)
File "/home/nikolai/venv/dask-gateway/lib/python3.9/site-packages/distributed/comm/addressing.py", line 171, in resolve_address
backend = registry.get_backend(scheme)
File "/home/nikolai/venv/dask-gateway/lib/python3.9/site-packages/distributed/comm/registry.py", line 81, in get_backend
raise ValueError(
ValueError: unknown address scheme 'gateway' (known schemes: ['inproc', 'tcp', 'tls', 'ucx', 'ws', 'wss'])
```

**Anything else we need to know?**:

Noticed the issue in a kubernetes-deployed setup on google cloud, but it is reproducible with the local installation as well.

The code works with a distributed cluster, e.g.

```python
from dask.distributed import Client, Queue
client = Client()
q = Queue("log")

def log_something():
q.put(42)

client.submit(log_something).result()
q.get()
```

**Environment**:

- Dask version: `2021.6.2`,
- dask-gateway version: `0.9.0`
- Python version: `3.9.5`
- Operating System: arch linux
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.