upload directories
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Sometimes I'm working on a python package which is just a directory with contents constantly being altered. I just want to send that directory to my dask workers without making a zipfile or egg or whatever. Currently I use these functions to get the job done:
def fn_to_targz_string(fn):
with io.BytesIO() as bt:
with tarfile.open(fileobj=bt,mode='w:gz') as tf:
tf.add(fn,arcname=os.path.basename(fn))
bt.seek(0)
s=bt.read()
return s
def extract_targz_string(s,*args,**kwargs):
import io,tarfile
with io.BytesIO() as bt:
bt.write(s)
bt.seek(0)
with tarfile.open(fileobj=bt,mode='r:gz') as tf:
tf.extractall(*args,**kwargs)
If we used tarfile in this way for Client.upload_file, it wouldn't matter whether the user wanted to upload a file or a directory. It should just work.
I believe the only necessary changes would be replacing the beginning of `Client._upload_file` with something like `fn_to_targz_string` (shown above) and the beginning of `Worker.upload_file` with something like `extract_targz_string` (shown above). If you wanted to be complete, you could also add something to the belly of `Worker.upload_file` so that the module would be properly inserted into `names_to_import`.
Contributor guide
Assessment
This issue has not been assessed yet.