CI: Mako performance test for PRs?
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
According to @kaomakino , Mako test takes only 2-3 hours to finish on a c5d.9xlarge VM; and the test can be optimized to run faster.
Although nightly performance test (Mako and circus) can help catch performance regression daily, it seems still not-ideal when we have many (more than 5) PRs merged on a day near release.
If we can run a small-scale performance test on specified PRs, we can catch performance regression faster and earlier, which will save engineer time.
Since we have already run 100K correctness on almost each PR, which takes longer time and more resources to finish, computation resource cost seems not an issue.
Things to discuss:
- [ ] What do people think about this idea? Do you want such a test run on your PR?
- [ ] How complex is it to implement this?
Contributor guide
Research direction
The issue discusses the existing nightly Mako and circus performance tests and the 100K correctness test used on pull requests. Start by locating those CI jobs and their configuration, then determine how a smaller performance run could be selected for specified PRs. Done means an agreed implementation exists for triggering and evaluating that PR-specific test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ci-cd, performance, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100