oxidecomputer / oxidecomputer/omicron

Slow incremental build/test workflow.

Open
#4,009 21 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

build development
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

tl;dr; A single-line change in a saga has a 10-minute build penalty for running tests. I'm developing on a 64-core machine with 256G of RAM using NVMe disks with trim enabled.

I run tests with cargo nextest run. I do not typically use -p as it's not always clear what packages my changes may intersect with. In these cases, I feel like the build system should be the one figuring out what needs to be rebuilt and not the user. It's also not clear why a change in a saga, which is seemingly near the top of the build pyramid, would cause a rebuild of unrelated testing packages.

Here is what a test run looks like with no-compilation, e.g. just running the tests.

real     3:32.627233210
user  3:35:25.998393895
sys   1:11:30.318353845
trap     2:22.183659850
tflt        4.635349996
dflt       47.150890796
kflt        0.837772914
lock 1751:11:53.945211231
slp  36:06:41.425071578
lat   3:53:12.177588657
stop  3:41:12.590832239

Here is a one-line change to saga code.

real    14:04.030041228
user  4:02:55.102264210
sys   1:07:44.130983296
trap     2:29.129798646
tflt        2.271531121
dflt       31.685564198
kflt        0.299503476
lock 1771:59:46.664672742
slp  38:27:07.709200291
lat   4:01:12.321380386
stop  3:54:09.085315193

Within that time the build time is the majority as reported by cargo.

Finished test [unoptimized + debuginfo] target(s) in 10m 29s

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported comparison with cargo nextest run, using the one-line saga change and the no-compilation baseline described in the issue. Inspect Cargo's reported build work and dependency decisions; done means identifying why unrelated testing packages rebuild and demonstrating a materially shorter incremental build.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system, performance, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.