boltpkg / boltpkg/bolt

Chunks

Open
#199 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2.4k
Forks
97
PR merge metrics
No merged PRs in 30d

Description

There are generally a couple different ways to run tools in a Bolt repo:

- Globally (across all workspaces at once)
- By Workspace
- Parallel
- Chunks by dependency tree
- Serial

When you have a tool that could either run globally or by workspace, you have to
make a tradeoff decision:

- Does the tool take a long startup time? By workspace is probably slower, having to startup over and over.
- Does the tool have a long process time? Globally is probably slower, would benefit from parallelization.

Right now we don't offer anything in between these two extremes. The different
"by workspace" modes don't help at all because they all spawn a new process for
every workspace.

```
(./) $ tool "./packages/*"
```

```
(./packages/a) $ tool "."
(./packages/b) $ tool "."
(./packages/c) $ tool "."
(./packages/d) $ tool "."
```

But what if we could break workspaces into "chunks" and spawn a smaller number
of processes?

```
(./) $ tool "./{packages/a,packages/b}"
(./) $ tool "./{packages/c,packages/d}"
```

This could allow us to increase parallelism of a tool while also reducing the
number of processes that it starts up.

## CLI

I would focus on just when the `--parallel` flag is on initially.

```
bolt workspaces exec --parallel --chunks -- tool "()"
```

If you wanted to limit the number of items to place in a chunk at a time:

```
bolt workspaces exec --parallel --chunks --max-chunk-size 10 -- tool "()"
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.