letsencrypt / letsencrypt/boulder

mtca: crash recovery

Open
#8,902 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.8k
Forks
649
Avg merge
3d 23h
Merged PRs (30d)
24

Description

The MTCA should gracefully handle crashes or errors during sequencing. Here's a sketch of what that should look like:

During sequencing:

  1. Calculate the latest frontier (including root hash and tree size).
  2. Flush that frontier to a pending area (possibly pending/<treesize>-<roothash>[^1]).
  3. Sign a checkpoint for the new root hash.
  4. Store the signature in the DB.
  5. Copy the pending tiles into the live area. Allow overwrites and retries; each copy operation can be retried as many times as necessary until all succeed.
  6. Publish the checkpoint as a signed note.
    If we fail anytime before (2) succeeds, we abandon the stored tiles and try again later (TODO: what if trying again later results in the same treesize and roothash?).

If we fail after (2) succeeds, we reconstruct the frontier from the pending area, check that it matches the expected root hash, and proceed from step (3). If there's no stored signature we will re-sign the checkpoint, but since we know the tree state that we intended to issue, this does not create a fork, even if we previously created a signature for the same tree state and then crashed.

Overall flow inspired by https://github.com/FiloSottile/sunlight/issues/11 and https://github.com/FiloSottile/sunlight/commit/a481639d3853ae7e185dc07190528a6b9374c278.

[^1]: So that in the bad situation where conflicting MTCAs are running, they conflict at the DB-write step rather than stepping on each others' tiles.

Part of #8738

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the MTCA sequencing entry point and the database and tile-persistence code that correspond to the numbered flow. Read how frontier state, signatures, pending tiles, live tiles, and published checkpoints are currently handled. Done means crashes at each stage can resume safely without inconsistent or conflicting tree state, with tests covering recovery and retry behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.