Improve CI testing
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
There are a few things which currently slow down our CI times, which cause long CI times and potential cases where CI times out as certain steps (such as testing) take too long. We can improve this with a few changes.
- `Run unit tests` step runs all test projects always. This is a bit excessive as changing something in the Storage tools shouldn't require the entire repos unit tests to run. This step should be scoped to the projects changed (unless the projects changed are Core libraries or `/eng`, in which case the current behavior is correct).
- `Run unit tests` and `Run recorded tests` should be merged into a single `Run tests` step. The two steps were kept separate when `UnitTests` and `LiveTests` were different projects. These have been merged and other infrastructure has been improved so that these no longer need to be separate things. (The first point above should be completed first before this is done)
- Each live test class results in a new Test Proxy being started. This may need to be kept but it also means that every live test class starts an application that isn't cheap to maintain. This also results in more resource contention in the runners.
- Improve test run outputs to be more actionable. Right now, most output is just indicating that tests passed or is a few large JSON blobs. This makes troubleshooting which tests failed and why much more difficult. Test logging output should be cleaned up.
Contributor guide
Research direction
Start by locating the CI definition containing the `Run unit tests` and `Run recorded tests` steps, then review how changed projects and Core or `/eng` changes are detected. Check the existing test-project and Test Proxy setup before assessing the step merge, proxy reuse, and logging changes. Done means CI runs the appropriate tests, combines the test steps, avoids unnecessary proxy startup where possible, and produces actionable failure output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- ci-cd, testing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100