dask / dask/distributed

client.submit not working with client.upload_file after first time

Open
#3,739 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Below are my local and cloud Dask version.

Dask version: 2.14
Distributed version: 2.14

We have a production Dask cluster in GKE, and shared among our engineers. Now, we want to create a python script for boilerplate purpose, simply name, `boilerplate.py`,

```python
def inc(x):
if x > 10:
raise Exception('x > 10')
return x + 1
```

And our client script,
```python
from dask.distributed import Client
import boilerplate as bp
bootstrap = # proxied
client = Client(bootstrap)
client.upload_file('boilerplate.py')

client.submit(bp.inc,7).result() # no error
client.submit(bp.inc,11).result() # got error
```

Now we fixed `boilerplate.py`,

```python
def inc(x):
if x > 20:
raise Exception('x > 20')
return x + 1
```

And rerun client script to reupload fixed `boilerplate.py`,
```python
from dask.distributed import Client
Exception: x > 10
bootstrap = # proxied
client = Client(bootstrap)
client.upload_file('boilerplate.py')

client.submit(bp.inc,7).result() # no error
client.submit(bp.inc,11).result() # got error
```

It is still same exception,
```
Exception: x > 10
````

The problem here, we cannot restart our Dask cluster, it might disturb other important tasks. It might seems got some caching layer here, this is only happened if we use `file_upload` + `submit`, if we use `file_upload` + `run`, it use latest update `boilerplate.py`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.