oxidecomputer / oxidecomputer/humility

humility test suite could be less needlessly cruel

Open
#451 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
584
Forks
63
Avg merge
3h
Merged PRs (30d)
2

Description

Running the Humility test suite is one of the most unkind things I've done to my poor laptop: it takes about five minutes to run, and pins every CPU core at 100% for the entire duration:
image

It seems like this is because the complexity of the test suite is multiplicative: it's the number of cores times the number of commands we run. This means that adding a new command increases the total size of the test suite by len(cores), and adding a new core increases the total size of the test suite by len(commands).

There's probably a way to make this a bit less of a brutal dev machine torture test: in at least some cases, such as counters, a command is only "interesting" to run on newer cores that contain the structures the command interprets. Running humility counters on any core except for the one I added specifically to test the counters will just result in humility error: no counters found. So, we probably don't need to run the several different counters commands in the test suite against every core we currently test with. If we added a notion of "only run this command against these cores" to the test framework, we could make it a little less painful to run.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the Humility test suite and locating the test framework that forms the command-by-core matrix. Examine how commands such as humility counters behave on different cores. Done means commands can be limited to relevant cores while the existing coverage remains intact and the suite avoids redundant runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.