IntersectMBO / IntersectMBO/ouroboros-consensus
Avoid header validation during block validation
- Dominant language
- Haskell
- Stars
- 67
- Forks
- 43
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 43
Description
We recently introduced the concept of header revalidation to mitigate Shelley's slow header validation: input-output-hk/ouroboros-network#2521.
Currently, we first download headers via the ChainSync protocol, validate them (expensive), the BlockFetch client then downloads the blocks corresponding to these headers, adds them to the ChainDB, which validates these blocks, *including their headers*. This means we revalidate each header *twice*, which is redundant and, worse, in the critical path of syncing.
We can avoid this by letting `ChainDB.addBlock` use `revalidateHeader` instead of `validateHeader` (some refactoring is probably required to make this work). `ChainDB.addBlock` should have as precondition that the header of the block is valid, which will be the case for all blocks added by the BlockFetch client.
However, we do add blocks of which we *haven't validated the headers yet* in one place, i.e., when adding a self-forged block. We can provide a variant of `ChainDB.addBlock` for this.
---
A rejected idea I had to avoid this distinction: introduce a separate cache of previously validated header hashes, similar to the cache of previously validated block hashes. It would be maintained by the ChainDB and the ChainSync client would add hashes to it. When the hash is already in the validated block cache, it doesn't have to be added to the validated header cache. When a block passes validation, its hash can be removed from the validated header cache and insert in the validated block cache. The validated header cache would be GC'ed similarly to how the validated block cache is GC'ed.
---
An idea @dcoutts had: if we have a previously validated header cache, we can use it to avoid validating the same header by different ChainSync threads. However, we have to be careful:
> If I have 20 upstream peers, and normally I scale the CPU use for validating 1 header (and reusing it 19 times) then when my peers attack me I now have a 20x CPU use spike, and I don't have the resources. And this is all under the control of the attacker.
Duncan proposes that we only do this optimisation when bulk syncing. This is a separate optimisation that we can consider later.
Contributor guide
Assessment
This issue has not been assessed yet.