etcd-io / etcd-io/etcd

Mixed-version robustness test failing

Open
#21,908 4 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
2d 21h
Merged PRs (30d)
43

Description

This is an AI assisted analysis, robustness tests look to have been broken https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/logs/ci-etcd-robustness-main-amd64 since the raft bump and https://github.com/etcd-io/etcd/pull/21846 and https://github.com/etcd-io/etcd/pull/21857 has not fully fixed them.

### Analysis

`ci-etcd-robustness-main-amd64` fails in mixed-version (3.6 + 3.7) runs when merging per-member WAL history. The bootstrap ConfChange entry at raft index 1 is reported as mismatching even though the members agree on the conf change. The only differing bytes are `\b\x00` (`ConfChange.Id` encoded as 0), which a 3.6 member writes explicitly and a 3.7 member omits.

Run: https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-robustness-main-amd64/2062255973065560064

```
EtcdHighTraffic/ClusterOfSize3/MemberReplace (etcd 3.6.12 + 3.7.0-beta.0):
main_test.go:113: mismatching entries on raft index 1, diff:
"Data": bytes.Join({
- "\b\x00",
"\x10\x00\x18...{"id":12416079282240904009,...}
}, ""),
"Type": s"EntryConfChange",
```

Note: prow shows this as flaky because `--count 120` reruns the same scenario names and only some iterations hit the mixed-version MemberReplace path. The failure is deterministic on that path.

Contributor guide

Open the contributing guide

Research direction

Start at main_test.go:113 and reproduce the mixed-version MemberReplace failure with the linked ci-etcd-robustness-main-amd64 run. Compare the raft index 1 ConfChange entries written by etcd 3.6 and 3.7, then verify the mixed-version robustness scenario completes without a mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.