oxidecomputer / oxidecomputer/omicron

CI speed should be more observable

Open
#4,471 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I'm going to inspect the output of 03c7f12cf7a70ed7b360a310e7a6b46d0075ac72 , with a breakdown of "where are we spending our time".

I would like to automate the duration and tracking of each of these steps, so it's more obvious where optimization efforts should go.

Tasks:

  • Identify the steps involved with the following jobs
    • build-and-test
    • build OS images
    • build TUF repo
  • Alter the jobs to emit more information summarizing "where time was spent"
  • Start tracking this information over time

Breakdown

  • build-and-test(helios): 1h 28m 19s
  • build-and-test(ubuntu-22.04): 52m 33s
  • buildomat / helios / build OS images: 1h 22m 53s
  • buildomat / helios / build TUF repo: 1h 56m 14s

There are other tasks here, but this is meaty enough to dig into and start optimizing.

build-and-test (helios)

Time Seq Description How long? Percentage of Total Time
2023-11-08T14:46:46.396Z 1 Job Start, doing minimal setup 4.197 seconds 0.08%
2023-11-08T14:46:50.593Z 15 Download cargo, rust (1.72.1) 25.198 seconds 0.48%
2023-11-08T14:47:15.791Z 59 Authentication 0.043 seconds 0.001%
2023-11-08T14:47:15.834Z 61 Clone and checkout 7.026 seconds 0.13%
2023-11-08T14:47:22.860Z 77 Simply running cargo --version downloads another rust toolchain (1.73.0), and then we download nextest too 25.275 seconds 0.48%
2023-11-08T14:47:48.135Z 121 Using pkg to download prerequisites (build-essential, llvm, xml, etc) 96.942 seconds 1.85%
2023-11-08T14:49:25.077Z 263 Download cockroach, clickhouse, console, maghemite, dendrite, tranceiver control 35.246 seconds 0.673%
2023-11-08T14:50:00.323Z 713 Downloading third-party dependency crates 35.931 seconds 0.686%
2023-11-08T14:50:36.254Z 1435 Compiling third party crates 600.307 seconds 11.47%
2023-11-08T15:00:36.561Z 3663 Compiling our crates 2148.442 seconds 40.69%
2023-11-08T15:36:05.003Z 4024 Start running nextest, which appears to be downloading and compiling more 3P crates 13.441 seconds 0.257%
2023-11-08T15:36:18.444Z 4233 Run crdb-seed (which appears to also compile from scratch ) 232.27 seconds 4.44%
2023-11-08T15:40:10.714Z 4562 Run crdb-seed 6.455 seconds 0.123%
2023-11-08T15:40:17.169Z 4571 Run tests 1897.63 seconds 36.277%
2023-11-08T16:11:54.799Z 5491 Build doctest 65.025 seconds 0.096%
2023-11-08T16:11:59.824Z 6315 Run doctest 5.025 seconds 2.244%
2023-11-08T16:13:57.214Z 6861 Finished running doctest N/A N/A

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting commit 03c7f12cf7a70ed7b360a310e7a6b46d0075ac72 and the referenced build-and-test run. Identify the definitions for the build-and-test, build OS images, and build TUF repo jobs, then determine how each step can report duration. Done means the jobs summarize where time was spent and that information is tracked over time.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ci-cd, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.