apple / apple/foundationdb

CI: Mako performance test for PRs?

Open
#3,158 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

According to @kaomakino , Mako test takes only 2-3 hours to finish on a c5d.9xlarge VM; and the test can be optimized to run faster.

Although nightly performance test (Mako and circus) can help catch performance regression daily, it seems still not-ideal when we have many (more than 5) PRs merged on a day near release.

If we can run a small-scale performance test on specified PRs, we can catch performance regression faster and earlier, which will save engineer time.

Since we have already run 100K correctness on almost each PR, which takes longer time and more resources to finish, computation resource cost seems not an issue.

Things to discuss:
- [ ] What do people think about this idea? Do you want such a test run on your PR?
- [ ] How complex is it to implement this?

Contributor guide

Open the contributing guide

Research direction

The issue discusses the existing nightly Mako and circus performance tests and the 100K correctness test used on pull requests. Start by locating those CI jobs and their configuration, then determine how a smaller performance run could be selected for specified PRs. Done means an agreed implementation exists for triggering and evaluating that PR-specific test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ci-cd, performance, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.