IntersectMBO / IntersectMBO/cardano-ledger

Provide faster migration from old serialised ledger state when format changes

Open
#3,131 1 comment 0 reactions 0 assignees View on GitHub
:credit_card: technical-debt
Dominant language
Haskell
Stars
295
Forks
179
Avg merge
4d 7h
Merged PRs (30d)
29

Description

# Context

- The ledger state is periodically check-pointed on disk by the ChainDB component of consensus in order to provide faster boot of cardano-node: Reading and verifying serialised ledger state, even though it's > 2GB, is still several orders of magnitude faster than replaying the whole chain (about 2-3h)
- When the ledger state structure changes, there is a need to migrate this old format to a new format
- If the old format is somewhat compatible with the new one, this can be straightforward: Just read the old version and write a new one
- There are common cases where this is not possible nor straightforward because the updated ledger format requires data that's not in the old ledger format but is stored in the chain (see #3113 for an example). In this situation the current approach is to drop the old ledger state and replay the full chain with the new ledger

# Problem

1. Replaying the chain is slow and inefficient, it implies the new node won't be operating for a couple of hours
2. In the context of [Mithril](https://mithril.network) and more generally snapshotting the node's state, this strategy implies part of the benefits snapshotting provides will be lost should we try to restore a snapshot that requires replay. This is evidenced in the table provided [here](https://github.com/input-output-hk/mithril/wiki/Logbook-2022-H1#how-to-produce-snapshots-1) that shows a 6-7x factor between restoration with and without a proper ledger state)

# Solution

A possible approach would be to provide some _migration procedure_ or _migration code_ when a new version of the ledger is deployed that would take care of:

1. converting the existing serialised ledger state to the new format by decoupling the deserialisation process from the actual updated data structure
2. when needed, provide new data by walking or "reducing" the chain only taking into account the required data and not the full blown ledger rules

For example, in the case of #3113 a ledger being fed an old version of the ledger state would be able to complete the missing data (exact deposits) by going through all the blocks and only computing the `stake key -> deposit` distribution without taking care of all other rules. Once updated the new ledger state will be stored updated on disk and carry on with the chain sync process.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.