[Perf] Run all package versions of a test sequentially
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 143
Description
Currently, the perf automation runs all tests on a given set of package versions sequentially. This is done to minimize the number of builds, since sources are built in-place in the repo. The downside is that different package versions for a test might be run far apart in time, which increases variance.
An alternative would be to run all package versions of a test seqentially. The downside is that we either need to re-compile between every test run (which is slow), or we need to compile everything to output directories that can be run independently (which may be difficult or impossible in some languages).
For example, the current test order is:
* Download, 10KB, packages
* Download, 10MB, packages
* Upload, 10KB, packages
* Upload, 10MB, packages
* Download, 10KB, sources
* Download, 10MB, sources
* Upload, 10KB, sources
* Upload, 10MB, source
The new test order would be:
* Download, 10KB, packages
* Download, 10KB, sources
* Download, 10MB, packages
* Download, 10MB, sources
* Upload, 10KB, packages
* Upload, 10KB, sources
* Upload, 10MB, packages
* Upload, 10MB, source
Contributor guide
Assessment
This issue has not been assessed yet.