embulk / embulk/embulk

Command-line (ssh) distributed executor

Open
#30 0 comments 0 reactions 0 assignees View on GitHub
new feature
Dominant language
Java
Stars
1.8k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

Idea:
I want to execute Embulk using this command line:

```
$ embulk run config.yml --execute-command "ssh %(find_host %{index}) embulk run-task %{task} %{index}"
```

With this command, embulk executes the given ssh command for each task. Then `embulk run-task` executes the given task on another host.
This is the simplest way to run embulk on distributed environment. We don't have to install Hadoop, YARN, Mesos, etc.

Some distributed execution environment such as mpich2 or Sun Grid Engine may work using this command-line.

See also: #29

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.