oasisprotocol / oasisprotocol/oasis-core
CI: Introduce representative benchmarks pipeline
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 369
- Forks
- 151
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 6
Description
In the past there was daily benchmarks pipeline that collected metrics from the e2e tests and analyzed them for regression.
In practice it was ignored for a long time, likely because the load was non-representative.
Challenge
Find representative load:
- Hard to simulate real load unless on the testnet/mainnet.
- What is relevant load? Checkpoints sync, genesis sync or sync 3 months of data?
- The relevant load should test more then just storage sync...
- Automatic genesis sync would be nice to see if anything is breaking but this would take too much time.
- How performant are current CI runners? Probably we would need custom bare metal instance for it...
Proposed solution
Have a weekly pipeline or on release pipeline, that triggers representative load and also implicitly checks breaking changes.
E.g. sync 3 most recent months of the data and collect and compare the actual metrics, between the pipeline runs.
https://github.com/oasisprotocol/oasis-core/issues/6423 enables this, whilst https://github.com/oasisprotocol/oasis-core/issues/6422 may define an upper bound for the sync age.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing daily benchmarks pipeline and the e2e tests that currently collect metrics. Use issues #6423 and #6422 to understand the proposed sync capability and age limit, then define a representative load and whether it runs weekly or on releases. Done means the pipeline runs the selected sync, collects comparable metrics, and checks for regressions or breaking changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- blockchain, ci-cd, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100