microsoft / microsoft/CCF

Allow joiners without snapshot to reuse local ledger files

Open
#6,975 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
876
Forks
260
Avg merge
1d 11h
Merged PRs (30d)
157

Description

Following up discussion in #6968, I noticed a surprising duplication of ledger files:

> they create another copy of the ledger in their `.current` directories, even though they already have it in their `.committed` directories!

Investigating that eventually led to #3064, which suggests this is deliberate and expected behaviour:

> The consequence of this is that, when a node is started without a snapshot but with access to another ledger (e.g. via --read-only-ledger-dir), both directories will contain some identical ledger files (since the node will have to replay all historical transactions).

I think this is confusing, on top of being obviously inefficient and expensive.

I'd like to explore a few things around this:

- [ ] Confirm whether we should be calling `init_as_backup()` (and its resulting rollback) when joining without a snapshot. There are still several comments saying these paths should occur after a snapshot is applied.
- [ ] When initialising views and rolling back to 0 on this joiner, we should truncate all files from the `.current` directory. We're somehow in an odd situation where we claim to have done a rollback, but we haven't actually truncated the ledger files, leading to confusion when the replay reaches this point (see the original bug and fix).
- [ ] Try to make `ledger_append` use the existing (read-only) file where possible. Somewhere we decide we need a writeable file, so create a full copy, but deep inside `write_entry()` we already check if this is an entry we already hold, in which case this is an idempotent nop. We should strive for the latter behaviour.
- [ ] Can we replay the state from these ledger directories, as part of the join process? We'd replay if we had a snapshot, and we have to confirm through the consensus protocol where we _agree_ with the leader, but currently we always report that we're at 0 and receive the full history over consensus. We should be able to do some partial recovery-like replay process to reach a more helpful initial state.
- [ ] What happens when the read-only directory contains conflicting files, or a dead suffix? We have suggested operator processes to avoid this, but it's possible storage is temporarily unavailable (so recovery completes on a shorter ledger), and then returns (so a joiner has a longer ledger from the origin service, now read-only).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with discussion in #6968 and related #3064, then trace init_as_backup(), ledger_append, and write_entry() for a joiner using --read-only-ledger-dir. The issue spans rollback, file reuse, replay, and conflicting or dead suffix handling; it is done only once the intended behavior is decided and each applicable case is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.