google / google/comprehensive-rust

Speed up publish CI pipeline

Open
#2,767 16 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
33.4k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
10

Description

Currently a run of the [publish pipeline](https://github.com/google/comprehensive-rust/actions/workflows/publish.yml) is taking around 30 minutes.

Most of the time is spent in the "build all translation" step.

https://github.com/google/comprehensive-rust/blob/c1773a7fdbfc493f29005af6f3713745c6674d5d/.github/workflows/publish.yml#L55-L60

This is the main loop for building the translations. This is currently writing files into the `book//html` folder, then moved into `book/html/`. The build.sh script is heavily modifying the current state of the repository so this is not trivially parallelized in the same step.

Options:
- separate steps by using Githubs [job matrix](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow#using-a-matrix-strategy) to run each translation as a separate job. Note: this requires all other steps in that job to run multiple times which is wasting resources. Installing mdbook in that environment should be cached properly. The English translation should then also be handled like other languages as this is currently a special case.
- This also needs to take into account that the generated artifacts need to be used in later steps
- Alternative: Split the translation step into two steps (likely more efficient)
- (for each language) restore the entire book source as currently and copy all necessary files into a new `source//` directory
- With all sources in each `source/` folder, this can be easily parallelized after modifying the build script.

https://github.com/google/comprehensive-rust/blob/c1773a7fdbfc493f29005af6f3713745c6674d5d/.github/workflows/build.sh#L27-L28

![Image](https://github.com/user-attachments/assets/9901ef1b-b77f-4ba0-a06c-b0d7c2383c2c)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.