etcd-io / etcd-io/etcd

Support running on non-empty database

Open
#19,900 6 comments 0 reactions 0 assignees View on GitHub
area/robustness-testing priority/backlog type/feature
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
3d 3h
Merged PRs (30d)
42

Description

### What would you like to be added?

This task is a little harder as it requires diving into robustness linearization model, interested contributor beware!

Currently we are validating that database is empty before running tests https://github.com/etcd-io/etcd/blob/49767381e803cc45c308fabea3c636ebb20f1d2f/tests/antithesis/test-template/robustness/main.go#L96-L99.

This is done by reading revision of etcd before any traffic is sent and checking if revision is equal `1` https://github.com/etcd-io/etcd/blob/49767381e803cc45c308fabea3c636ebb20f1d2f/tests/robustness/traffic/traffic.go#L207-L227. Revision in etcd is a global logical clock for operations that start at 1. This ensures that there were no transactions executed on etcd before so etcd is empty.

This is also checked as first step of validation https://github.com/etcd-io/etcd/blob/49767381e803cc45c308fabea3c636ebb20f1d2f/tests/robustness/validate/validate.go#L113-L127. This validation is because this is an implicit assumption of the linearization code, that without this assertion would return hard to discern error.

We can try to improve the linearization to allow non empty code. So why linearization doesn't support non-empty database? Because it needs to a complete view of database, so it can simulate it. Assuming it's empty was just easier to implement. https://github.com/etcd-io/etcd/blob/49767381e803cc45c308fabea3c636ebb20f1d2f/tests/robustness/model/deterministic.go#L99-L108

To improve it we could just replace the Read of revision 1 at the beginning with operation to download the state of database and change the model to accept this state as initial one.

Steps;
1. We still need to make a special request at before all other requests. But now instead of checking `revision == 1`, we just need to read the database state. Changes in https://github.com/etcd-io/etcd/blob/49767381e803cc45c308fabea3c636ebb20f1d2f/tests/antithesis/test-template/robustness/main.go#L96-L99.
2. In validation we still need to confirm if the special Read request is present. However the conditions are different, we need to make sure it precedes all other requests that reads the whole database contents (or at least the prefix of all keys used in the robustness test). Changes in https://github.com/etcd-io/etcd/blob/49767381e803cc45c308fabea3c636ebb20f1d2f/tests/robustness/validate/validate.go#L113-L127.
3. We would want to add a way in [EtcdState](https://github.com/etcd-io/etcd/blob/49767381e803cc45c308fabea3c636ebb20f1d2f/tests/robustness/model/deterministic.go#L69-L75) to represent a state that etcd is not initialized, during which the [Step](https://github.com/etcd-io/etcd/blob/49767381e803cc45c308fabea3c636ebb20f1d2f/tests/robustness/model/deterministic.go#L111) method will only allow a read requests. We could hardcode the same request as in step 2.

### Why is this needed?

Allow multiple runs of antithesis locally.
/cc @henrybear327 @nwnt

Contributor guide

Open the contributing guide

Research direction

Start with tests/antithesis/test-template/robustness/main.go and tests/robustness/validate/validate.go to trace the initial database read and its validation. Then study EtcdState and Step in tests/robustness/model/deterministic.go to understand how the initial state is represented. Done means robustness validation and linearization accept a captured non-empty database state while preserving the required ordering of the special read.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.