Define Consensus QTAs and test that they're met

# The problem

We often find performance regressions at the system level when running our system-level benchmarks. Finding these regressions so late in the delivery pipeline has several pernicious effects on our release process and consumes significant developers' time:

- Releases are delayed due to regressions found too late.
- People from multiple teams are involved in investigating the regressions found.
- Performance regressions at the system level are hard to debug.

# A possible solution

By defining Quantitative Timeliness Agreements per-each Cardano component we can make sure that any potential performance regression is detected before changes are integrated downstream. This long-term goal concerns the definition of QTAs for consensus and validating that they are met.

In principle, we could use some of the observation points of the system-level benchmarks to define these QTAs (eg time required to take mempool snapshots, ticking the ledger, self-adopting a block, etc).

Consensus should prove that each QTA is met by running benchmarks.

The input data for the benchmarks could be Cardano mainnet. For new features, we might have to generate data.

Each QTA requires a time specification and might depend on an upstream component, such as the Ledger. This means that our benchmarks could detect regressions.

The metrics associated with each QTA are:

- timeliness (eg elapsed time).
- resources consumed (eg memory allocation, steps).

Typically, QTA metrics are instrumented by means of tracers.

After a benchmark, we might want to perform garbage collection to measure how much garbage was left over.


# Considerations when choosing which QTA benchmarks to implement

From the Consensus perspective there are two main categories of QTA:

1. *Upstream*, which are QTAs that we suggest to upstream components (eg because our design assumes them) that upstream benchmarks should be able to verify without any cooperation from us.
2. *1st line*, which are QTAs that Consensus-level benchmarks can verify much more easily than upstream benchmarks could. This kind of QTAs can be further subdivided according to whether they involve upstream code or not.

Examples of 1st line QTAs that involve upstream code are:

- Some key Querys (the Node team et al can help us identify these)
- Sync duration for a significant number of blocks
- Header validation duration
- Block validation duration (especially as of UTxO HD)
- Leadership check duration.
- The delay between a new-best new-longest block arriving and it being pipelined.

As for 1st line QTAs that do not involve upstream code it is not so clear if there are any, since basically everything in consensus involve at least one ledger state input.

We believe there is nothing that we do that should take a considerable amount of CPU: the bulk of this resource is consumed by crypto and ledger operations. However our component is the first layer that can use realistic data, and therefore we could facilitate tooling to prepare data for upstream components QTA benchmarks (eg Ledger and Crypto).

The fact that the Consensus layer uses comparatively less CPU than other components does not mean that we can't write sub-optimal code. A possible way of checking this is by counting invocations to Ledger and Crypto operations and corroborating that they match our expectations. As a simplistic example, if we validate `n` blocks one would expect `n` invocations to the `applyBlock` function.

Also, there were discussions about [the cost of adding an Era](https://github.com/input-output-hk/cardano-node/issues/4213). While we don't have strong evidence to support the claim that each Era increases the processing time in a significant way, we could develop a QTA benchmark to validate this.

# Epics

The following epics describe concrete steps we could take to improve our capability of detecting regressions as early as possible.

- [ ] Count invocations to ledger rules and crypto invocations when the node is syncing. We could use [this tool](https://github.com/input-output-hk/ouroboros-network/pull/4339) for this purpose.
- [ ] Count invocations to ledger rules and crypto invocations when the node is caught up. We could use the `ThreadNet` tests for this.
- [ ] [Use bench-ledger ops analysis for QTA tests.](https://github.com/input-output-hk/ouroboros-consensus/issues/79). This will give us a QTA that measures adoption time and also the components of the adoption procedure. If necessary we could develop a different benchmark that uses the node directly (using a simple enough setup). Setting the infrastructure for this will take a considerable amount of work. We will have to choose which segments to use for this (eg we might want to include epoch transition boundaries). We need to consider carefully which segments of data to use now, and also in the future.
- [ ] Prove that the hard-fork depth is inconsequential (when compared with the cost of ledger and crypto ops).
- [ ] Create tooling for upstream components to run their own QTA analysis.
- [ ] Leverage [new ThreadNet](https://github.com/input-output-hk/ouroboros-consensus/issues/651) tests for analyzing performance (TODO ask Javier for ideas).