dask / dask/distributed

upload directories

Open
#925 15 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Sometimes I'm working on a python package which is just a directory with contents constantly being altered. I just want to send that directory to my dask workers without making a zipfile or egg or whatever. Currently I use these functions to get the job done:

def fn_to_targz_string(fn):
with io.BytesIO() as bt:
with tarfile.open(fileobj=bt,mode='w:gz') as tf:
tf.add(fn,arcname=os.path.basename(fn))
bt.seek(0)
s=bt.read()
return s

def extract_targz_string(s,*args,**kwargs):
import io,tarfile
with io.BytesIO() as bt:
bt.write(s)
bt.seek(0)
with tarfile.open(fileobj=bt,mode='r:gz') as tf:
tf.extractall(*args,**kwargs)

If we used tarfile in this way for Client.upload_file, it wouldn't matter whether the user wanted to upload a file or a directory. It should just work.

I believe the only necessary changes would be replacing the beginning of `Client._upload_file` with something like `fn_to_targz_string` (shown above) and the beginning of `Worker.upload_file` with something like `extract_targz_string` (shown above). If you wanted to be complete, you could also add something to the belly of `Worker.upload_file` so that the module would be properly inserted into `names_to_import`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.