etcd-io / etcd-io/etcd

Robustness test linearization OOM

Open
#21,928 2 comments 0 reactions 1 assignee Claimed by @henrybear327 View on GitHub
type/bug
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
2d 21h
Merged PRs (30d)
43

Description

### Bug report criteria

- [x] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [x] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [x] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.

### What happened?

Robustness test nightly run jobs would sometime be terminated due to linearization validation OOM.

### What did you expect to happen?

Finish the linearization check without the job being killed due to OOM.

### How can we reproduce it (as minimally and precisely as possible)?

As we now save the report before validation https://github.com/etcd-io/etcd/pull/21807, we can now reproduce the issue locally.

A recent run https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-robustness-main-arm64/2063545174469906432, test case TestRobustnessExploratory_KubernetesHighTraffic_ClusterOfSize1_raftAfterSave=sleep, took about 40GB peak memory to validate the linearization.

### Anything else we need to know?

Looking at the traffic history and some initial testing using some ideas from https://github.com/etcd-io/etcd/issues/21864, I think it's going to be hard to eliminate the entire OOM situation due to the space exploration we are doing during the validation. But we now have some real test cases test against locally, thus, creating this tracking issue for future efforts.

### Etcd version (please run commands below)

```console
$ etcd --version
# paste output here

$ etcdctl version
# paste output here
```

### Etcd configuration (command line flags or environment variables)

# paste your configuration here

### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

```console
$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints= endpoint status -w table
# paste output here
```

### Relevant log output

```Shell

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.