google / google/wireit

Add a semantic granular way to specify the parallelism of dependencies

Open
#325 10 comments 5 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
6.4k
Forks
128
Avg merge
4d 10h
Merged PRs (30d)
4

Description

## Use case
- Two tasks that need to be run sequentially, and the order _may (or may not) matter_
- Ie. When two tasks modify the same output files

### Currently the way to make dependencies run in serial mode is the following
```json
"tasks": {
"dependencies": [
"task:one",
"task:two"
]
}
```

## 1. WIREIT_PARALLEL way
```
WIREIT_PARALLEL=1 npm run tasks
```
Which has the disadvantages of:
- It applies to the whole dependency graph
- Hard to type each time

## 2. Unix double ampersand way
```
npm run task:one && npm run task:two
```
Which has the disadvantages of:
- `&&` is a particular unix syntax and doesn't work on Windows
- This can be solved by introducing [npm-run-all's run-s](https://github.com/mysticatea/npm-run-all/blob/HEAD/docs/run-s.md) but requires a new dependency
- Disallows wire-it caching of the `tasks` itself if `tasks` has an aditional `command`, `input`, `output` fields
- Not wire-it native

---

# Suggested solution
Be able to define parallelism inside `package.json`
Don't apply it to the whole dependency graph, just apply to the command at hand

## Top-level Parallel
```json
"tasks": {
"parallel": 1,
"dependencies": [
"task:one",
"task:two"
]
}
```
Where `parallel` is a number.

## Top-level Concurrency
```json
"tasks": {
"concurrency": "sequential",
"dependencies": [
"task:one",
"task:two"
]
}
```
Where `concurrency` can be either `parallel` (default) or `sequential`.

## A mix of Parallel + Concurrency
```json
"tasks": {
"concurrency": "string | number",
"dependencies": [
"task:one",
"task:two"
]
}
```
Where `concurrency` can be either `parallel` (default), `sequential`, or a number specifying the exact number of parallelism

Now, this works for most common cases, but if we dig deeper, one might need more detailed control in order to optimize how the dependencies are run

## Dependency Groups
```json
"tasks": {
"concurrency": [
"sequential": ["task:one", "task:two"],
"parallel": ["task:a", "task:b", "task:c"],
"sequential": ["task:finish"]
],
"dependencies": [
"task:one",
"task:two",
"task:a",
"task:b",
"task:c",
"task:finish"
]
}
```
This will run
1. `task:one` and `task:two` first sequentially, then
2. `task:a`, `task:b`, `task:c` in parallel, and then
3. `task:finish`, in order.

What do you think?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.