IntersectMBO / IntersectMBO/ouroboros-consensus
Use the bench-ledger-ops analysis to create a benchmark
- Dominant language
- Haskell
- Stars
- 67
- Forks
- 43
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 43
Description
# Background
As part of [this long-term goal](https://github.com/input-output-hk/ouroboros-consensus/issues/72), we want to elaborate a benchmarking tool that, given two Consensus versions, it compares the cost of performing the 5 main ledger operations between said versions. These 5 ledger operations are:
1. Forecast.
2. Header tick.
3. Header application.
4. Block tick.
5. Block application.
These operations combined constitute the bulk of the time used for block adoption.
We want this tool to be usable in the development process.
# Motivation
We want to provide a means for the Consensus and Ledger developers, as well as the release engineers, to be able to spot performance regressions early on.
# Definition of done
Produce a tool that allows to compare the cost of the main ledger operations across two Consensus versions. This comparison can be carried out by inspecting the following artefacts produced by the tool. No automation in the detection of performance regression is required.
The tool should:
- Allow to specify the versions of Consensus to compare.
- Allow to specify the GHC to build a given Consensus version (to be compared)
- Allow to specify the RTS options to run `db-analyser`.
- Produce a plot per ledger operation, which shows the execution time of both versions (see [this example](https://github.com/input-output-hk/ouroboros-consensus/pull/161#issuecomment-1607147106)).
- TODO: Produce a report/table with and .
- Make each report traceable by storing data like "build information".
- Be properly documented so that other developers can use it.
- Yield results that are consistent with the system-level benchmarks.
Additionally, we should:
- Provide the developers with infrastructure (eg AWS instances) and data that they can use to run the benchmark comparison tool.
As future steps, we could consider running these benchmarks on CI, if that adds value.
# Subtasks
- [x] #132
- [x] #142
- [x] #153
- [x] #230
- [ ] #223
- [ ] Make beacon automatable:
- We’ve developed the existing prototype into an automatable, self-contained benchmark called `beacon`, as well as systematized workloads and run structure for it. Moreover, we’ve demonstrated the usefulness of the metrics and their reproducibility and identified domains that are viable for QTAs with system-level benchmarks.
- [x] #918
- [ ] #919
- [ ] Automate beacon runs:
- have runners (machines) with the same hardware
- make sure the runners do not execute any other process during benchmarks (they have exclusive access to that machine)
- run results should be stored
- developers can trigger runs manually.
- [ ] Integrate beacon runs with CI (beacon can be triggered via github actions, have means of retrieving data of those runs, API querying)
- [ ] #224
- [ ] Correlate system-level benchmarks (P&T) with the comparison tool data.
Contributor guide
Assessment
This issue has not been assessed yet.