Azure / Azure/azure-sdk-tools

[Perf] Run all package versions of a test sequentially

Open
#3,560 0 comments 0 reactions 1 assignee Claimed by @mikeharder View on GitHub
pillar-performance
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

Currently, the perf automation runs all tests on a given set of package versions sequentially. This is done to minimize the number of builds, since sources are built in-place in the repo. The downside is that different package versions for a test might be run far apart in time, which increases variance.

An alternative would be to run all package versions of a test seqentially. The downside is that we either need to re-compile between every test run (which is slow), or we need to compile everything to output directories that can be run independently (which may be difficult or impossible in some languages).

For example, the current test order is:

* Download, 10KB, packages
* Download, 10MB, packages
* Upload, 10KB, packages
* Upload, 10MB, packages
* Download, 10KB, sources
* Download, 10MB, sources
* Upload, 10KB, sources
* Upload, 10MB, source

The new test order would be:

* Download, 10KB, packages
* Download, 10KB, sources
* Download, 10MB, packages
* Download, 10MB, sources
* Upload, 10KB, packages
* Upload, 10KB, sources
* Upload, 10MB, packages
* Upload, 10MB, source

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.