oasisprotocol / oasisprotocol/oasis-core

CI: Introduce representative benchmarks pipeline

Open
#6,503 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
369
Forks
151
Avg merge
1d 3h
Merged PRs (30d)
6

Description

In the past there was daily benchmarks pipeline that collected metrics from the e2e tests and analyzed them for regression.

In practice it was ignored for a long time, likely because the load was non-representative.

Challenge

Find representative load:

  • Hard to simulate real load unless on the testnet/mainnet.
  • What is relevant load? Checkpoints sync, genesis sync or sync 3 months of data?
    • The relevant load should test more then just storage sync...
  • Automatic genesis sync would be nice to see if anything is breaking but this would take too much time.
  • How performant are current CI runners? Probably we would need custom bare metal instance for it...

Proposed solution

Have a weekly pipeline or on release pipeline, that triggers representative load and also implicitly checks breaking changes.

E.g. sync 3 most recent months of the data and collect and compare the actual metrics, between the pipeline runs.

https://github.com/oasisprotocol/oasis-core/issues/6423 enables this, whilst https://github.com/oasisprotocol/oasis-core/issues/6422 may define an upper bound for the sync age.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing daily benchmarks pipeline and the e2e tests that currently collect metrics. Use issues #6423 and #6422 to understand the proposed sync capability and age limit, then define a representative load and whether it runs weekly or on releases. Done means the pipeline runs the selected sync, collects comparable metrics, and checks for regressions or breaking changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
blockchain, ci-cd, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.