Command-line (ssh) distributed executor
Open
new feature
- Dominant language
- Java
- Stars
- 1.8k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
Idea:
I want to execute Embulk using this command line:
```
$ embulk run config.yml --execute-command "ssh %(find_host %{index}) embulk run-task %{task} %{index}"
```
With this command, embulk executes the given ssh command for each task. Then `embulk run-task` executes the given task on another host.
This is the simplest way to run embulk on distributed environment. We don't have to install Hadoop, YARN, Mesos, etc.
Some distributed execution environment such as mpich2 or Sun Grid Engine may work using this command-line.
See also: #29
Contributor guide
Assessment
This issue has not been assessed yet.