Azure / Azure/azure-sdk-for-rust

Consider supporting cargo-nextest

Open
#4,092 1 comment 0 reactions 0 assignees View on GitHub
EngSys test-enhancement
Dominant language
Rust
Stars
884
Forks
365
Avg merge
2d 19h
Merged PRs (30d)
109

Description

As we scale the number of crates in the repo, we'll start to hit some scalability issues. Among them is test run time. Anything that runs a bunch of tests - like `cargo test --workspace --no-fail-fast` - is going to be slow. [cargo-nextest](https://nexte.st) can help, but we'd need to solve a couple problems:

1. It's [configuration](https://nexte.st/docs/configuration/) is centralized. While we can use test patterns to apply defaults for a profile (like a "ci" profile), there's no easy way to build a hierarchy of configuration files like, say, `.gitignore`).
2. This runs more tests in parallel, which we could also accomplish with #4091; however, in local testing I found it not only breaks a bunch of tests, but has some persistent effect:
```bash
cargo test --no-fail-fast # works
cargo nextest run --no-fail-fast # a bunch of tests fail - mostly recorded but, IIRC, not all recorded
cargo test --no-fail-fast # fails, and from a small sample, it's all the same tests as the previous step
```

We also have to make sure `cargo test` keeps working because that's idiomatic and `rustup` nor `cargo` have any keep way to preinstall third-party dependencies like some languages' package managers do (there are some, but they have to be installed first: chicken/egg problem). We could recommend it in `CONTRIBUTING.md`, but then use it in our PRs and CIs with profiles as needed.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the listed cargo test and cargo nextest commands, including the persistent failures after nextest runs. Read CONTRIBUTING.md and compare the proposed configuration and CI/PR usage with #4091. Done requires a decided approach that preserves cargo test, addresses configuration and test isolation, and specifies any recommendation or CI integration.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.