letsencrypt / letsencrypt/boulder
mtca: crash recovery
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.8k
- Forks
- 649
- Avg merge
- 3d 23h
- Merged PRs (30d)
- 24
Description
The MTCA should gracefully handle crashes or errors during sequencing. Here's a sketch of what that should look like:
During sequencing:
- Calculate the latest frontier (including root hash and tree size).
- Flush that frontier to a
pendingarea (possiblypending/<treesize>-<roothash>[^1]). - Sign a checkpoint for the new root hash.
- Store the signature in the DB.
- Copy the
pendingtiles into thelivearea. Allow overwrites and retries; each copy operation can be retried as many times as necessary until all succeed. - Publish the checkpoint as a signed note.
If we fail anytime before (2) succeeds, we abandon the stored tiles and try again later (TODO: what if trying again later results in the same treesize and roothash?).
If we fail after (2) succeeds, we reconstruct the frontier from the pending area, check that it matches the expected root hash, and proceed from step (3). If there's no stored signature we will re-sign the checkpoint, but since we know the tree state that we intended to issue, this does not create a fork, even if we previously created a signature for the same tree state and then crashed.
Overall flow inspired by https://github.com/FiloSottile/sunlight/issues/11 and https://github.com/FiloSottile/sunlight/commit/a481639d3853ae7e185dc07190528a6b9374c278.
[^1]: So that in the bad situation where conflicting MTCAs are running, they conflict at the DB-write step rather than stepping on each others' tiles.
Part of #8738
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the MTCA sequencing entry point and the database and tile-persistence code that correspond to the numbered flow. Read how frontier state, signatures, pending tiles, live tiles, and published checkpoints are currently handled. Done means crashes at each stage can resume safely without inconsistent or conflicting tree state, with tests covering recovery and retry behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100