client.submit not working with client.upload_file after first time
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Below are my local and cloud Dask version.
Dask version: 2.14
Distributed version: 2.14
We have a production Dask cluster in GKE, and shared among our engineers. Now, we want to create a python script for boilerplate purpose, simply name, `boilerplate.py`,
```python
def inc(x):
if x > 10:
raise Exception('x > 10')
return x + 1
```
And our client script,
```python
from dask.distributed import Client
import boilerplate as bp
bootstrap = # proxied
client = Client(bootstrap)
client.upload_file('boilerplate.py')
client.submit(bp.inc,7).result() # no error
client.submit(bp.inc,11).result() # got error
```
Now we fixed `boilerplate.py`,
```python
def inc(x):
if x > 20:
raise Exception('x > 20')
return x + 1
```
And rerun client script to reupload fixed `boilerplate.py`,
```python
from dask.distributed import Client
Exception: x > 10
bootstrap = # proxied
client = Client(bootstrap)
client.upload_file('boilerplate.py')
client.submit(bp.inc,7).result() # no error
client.submit(bp.inc,11).result() # got error
```
It is still same exception,
```
Exception: x > 10
````
The problem here, we cannot restart our Dask cluster, it might disturb other important tasks. It might seems got some caching layer here, this is only happened if we use `file_upload` + `submit`, if we use `file_upload` + `run`, it use latest update `boilerplate.py`.
Contributor guide
Assessment
This issue has not been assessed yet.