erigontech / erigontech/erigon

Integration tests for erigondb.toml multiple reloading codepaths

Open
#19,588 1 comment 0 reactions 0 assignees View on GitHub
ErigonDB github_actions QA
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

this is a follow up from: https://github.com/erigontech/erigon/pull/19587

In the past, the DB settings (as of now, stepsize and max merge steps) used to be a hardcoded setting. But we quickly moved for exposing them as a hardcoded parameterized setting for experiments, and then now to push it to production on 3.4.

Problem: there are now multiple code paths and legacy vs new sync configurations scenarios, summarizing them here:

- Legacy datadir (3.3), no `erigondb.toml`, needs to assume legacy settings on migration and force write that file
- new sync on 3.4: the setting will come from downloader during the header-chain phase.
- already synced node (doesn't matter if from 3.3 or 3.4), erigondb.toml already exists, respect such setting
- `--no-downloader` will write `erigondb.toml` at start using the embedded hardcoded setting.

the critical scenario is the (2), since we can't assume any setting until it is downloaded and hot reloaded. assume the process can be interrupted between process launch and header-chain download.

also, in order to support such hot reloading, I had to instrument SharedDomains to allow such thing and make it a little flexible since some checkings are done at object construction (i.e. had to mock instead of assuming 0).

it all works now, but it seems to me that such settings lifecycle has become more "important" and fragile (anyone who touch ottersync lifecycle code can potentially break it). in the past downloader was quite a "dumb" process.

I think now we need to build some protection around it in the form of integration tests. Ideally we would have automated tests for all scenarios above using a real chain, but that's not scalable, the real chains are ever growing databases.

I'd like to propose to support some synthetic chain with shorter steps sizes, such we can have it already constructed and deployed in some test R2 bucket and use it to test those lifecycles.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.