Add option to distribute files to workers
- Dominant language
- Python
- Stars
- 69
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
From https://github.com/dask/dask-yarn/issues/49#issuecomment-456645866.
I think this would look like:
- A `worker_files` kwarg, which takes a mapping to forward to the `files` kwarg in the skein specification.
- A `worker-file` and `worker-archive` option in the cli. We could mirror spark's api here and have the localized name be optionally specified with a `#` separator (see https://spark.apache.org/docs/latest/running-on-yarn.html#important-notes). I'm not a huge fan of this api, but it is used by other tools and that might be argument enough to use it.
```
dask-yarn submit \
--worker-file /local/file/path \
--worker-file /other/local/file/path#name-in-container
```
Contributor guide
Research direction
Start by tracing the dask-yarn submit entry point and how its worker specification is assembled, then compare that with the skein specification's files field. Implement the worker_files kwarg and the worker-file and worker-archive CLI options, including optional # container names, and verify both interfaces forward the requested files to workers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100