dask / dask/dask-yarn

Add option to distribute files to workers

Open
#50 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
69
Forks
40
PR merge metrics
No merged PRs in 30d

Description

From https://github.com/dask/dask-yarn/issues/49#issuecomment-456645866.

I think this would look like:

- A `worker_files` kwarg, which takes a mapping to forward to the `files` kwarg in the skein specification.
- A `worker-file` and `worker-archive` option in the cli. We could mirror spark's api here and have the localized name be optionally specified with a `#` separator (see https://spark.apache.org/docs/latest/running-on-yarn.html#important-notes). I'm not a huge fan of this api, but it is used by other tools and that might be argument enough to use it.

```
dask-yarn submit \
--worker-file /local/file/path \
--worker-file /other/local/file/path#name-in-container
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing the dask-yarn submit entry point and how its worker specification is assembled, then compare that with the skein specification's files field. Implement the worker_files kwarg and the worker-file and worker-archive CLI options, including optional # container names, and verify both interfaces forward the requested files to workers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.