erigontech / erigontech/erigon

db/state: merge schedule non-determinism across nodes — impacts decentralized snapshot distribution

Open
#20,531 7 comments 0 reactions 0 assignees View on GitHub
ErigonDB
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

## Problem

The aggregator merge schedule is not fully deterministic across nodes. Two nodes at the same step can have different file layouts due to:

1. **Preverified legacy accommodation**: `getMergeLimit()` in `snap_repo.go` adjusts merge limits based on preverified files, which differ across binary releases
2. **Cross-domain coordination**: commitment domain must wait for accounts/storage to merge first. Interrupted merges leave divergent intermediate states
3. **Two merge code paths**: `aggregator.go` (old, active) vs `forkable_agg.go`/`snap_repo.go` (new) with different merge logic

### Why this matters

Decentralized snapshot distribution (#19660) requires nodes to compare what files they have. If the merge schedule is deterministic, two honest nodes at the same step produce identical files with identical torrent hashes — enabling direct hash comparison and threshold consensus (#19658).

If non-deterministic, we need structural range-coverage comparison instead of hash comparison, which is significantly more complex and weakens the trust model.

### Current merge schedule

The bit-manipulation schedule (`endStep & -endStep`) is inherently deterministic:

```
Step 8: [0-8)
Step 12: [0-8) [8-12)
Step 16: [0-16)
Step 24: [0-16) [16-24)
Step 32: [0-32)
```

But it's perturbed by:
- `maxSpan` cap from `stepsInFrozenFile` (configurable via `erigondb.toml`)
- `getMergeLimit()` returning larger sizes when preverified files exist
- Commitment domain holding merges until accounts/storage catch up

### Example of divergence

Node A (clean run):
```
v1.0-accounts.0-4096.kv (deep merge)
v1.0-accounts.4096-4224.kv (128 steps)
v1.0-accounts.4224-4232.kv (8 steps)
```

Node B (interrupted merge, restarted):
```
v1.0-accounts.0-2048.kv (partial deep merge)
v1.0-accounts.2048-4096.kv
v1.0-accounts.4096-4224.kv
v1.0-accounts.4224-4232.kv
```

Both cover steps 0-4232 but with different file boundaries and torrent hashes.

### Proposed resolution

Make the merge schedule purely deterministic: given `(step, MergeStages config)`, the expected file layout is computable. On startup, identify missing merges and execute them to converge to the canonical layout.

This would mean:
- All nodes at the same step produce identical files
- chain.toml can use hash comparison (simple, torrent-compatible)
- Threshold consensus works (same hash = agreement)
- UCAN can sign `(chain, step)` tuples — layout is implied

### Alternative

Accept non-determinism and use structural range-coverage comparison in chain.toml. Files are compared by step range, not hash. More complex, weaker trust model.

### Blocking

This decision blocks the chain.toml V2 format design (#19660) — the format differs depending on whether we can assume deterministic layouts.

## Related issues

- #19660 — Decentralized snapshot distribution (umbrella)
- #19657 — POC
- #19658 — Statistical trust (threshold consensus)
- #19659 — Identity trust (UCAN)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.