etcd-io / etcd-io/etcd

panic when two snapshots are received in short period

Open
#18,055 13 comments 0 reactions 1 assignee Claimed by @darshil929 View on GitHub
area/robustness-testing help wanted stage/tracked type/bug
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
2d 21h
Merged PRs (30d)
43

Description

### Bug report criteria

- [ ] This bug report is not security related, security issues should be disclosed privately via [etcd maintainers](mailto:etcd-maintainers@googlegroups.com).
- [ ] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [ ] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [ ] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.

### What happened?

https://github.com/etcd-io/etcd/actions/runs/9188844320/job/25269542381
```
{"level":"panic","ts":"2024-05-22T10:31:40.994846Z","caller":"etcdserver/server.go:1010","msg":"failed to open snapshot backend","error":"failed to find database snapshot file (snap: snapshot file doesn't exist)","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applySnapshot\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1010\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:947\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func6\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:840\ngo.etcd.io/etcd/pkg/v3/schedule.job.Do\n\tgo.etcd.io/etcd/pkg/v3@v3.6.0-alpha.0/schedule/schedule.go:41\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob\n\tgo.etcd.io/etcd/pkg/v3@v3.6.0-alpha.0/schedule/schedule.go:206\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/v3@v3.6.0-alpha.0/schedule/schedule.go:187"}
2024-05-22T10:31:41.0044258Z /home/runner/actions-runner/_work/etcd/etcd/bin/etcd (TestRobustnessExploratoryKubernetesHighTrafficClusterOfSize3-test-0) (40381): {"level":"info","ts":"2024-05-22T10:31:40.994887Z","caller":"etcdserver/server.go:984","msg":"applied snapshot","current-snapshot-index":1590,"current-applied-index":1590,"incoming-leader-snapshot-index":1857,"incoming-leader-snapshot-term":2}
2024-05-22T10:31:41.0060472Z /home/runner/actions-runner/_work/etcd/etcd/bin/etcd (TestRobustnessExploratoryKubernetesHighTrafficClusterOfSize3-test-0) (40381): {"level":"panic","ts":"2024-05-22T10:31:40.994904Z","caller":"schedule/schedule.go:202","msg":"execute job failed","job":"server_applyAll","panic":"failed to open snapshot backend","stacktrace":"go.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob.func1\n\tgo.etcd.io/etcd/pkg/v3@v3.6.0-alpha.0/schedule/schedule.go:202\nruntime.gopanic\n\truntime/panic.go:770\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\tgo.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\tgo.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber.org/zap.(*Logger).Panic\n\tgo.uber.org/zap@v1.27.0/logger.go:285\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applySnapshot\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1010\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:947\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func6\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:840\ngo.etcd.io/etcd/pkg/v3/schedule.job.Do\n\tgo.etcd.io/etcd/pkg/v3@v3.6.0-alpha.0/schedule/schedule.go:41\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob\n\tgo.etcd.io/etcd/pkg/v3@v3.6.0-alpha.0/schedule/schedule.go:206\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/v3@v3.6.0-alpha.0/schedule/schedule.go:187"}
2024-05-22T10:31:41.0074671Z /home/runner/actions-runner/_work/etcd/etcd/bin/etcd (TestRobustnessExploratoryKubernetesHighTrafficClusterOfSize3-test-0) (40381): panic: failed to open snapshot backend [recovered]
2024-05-22T10:31:41.0077589Z /home/runner/actions-runner/_work/etcd/etcd/bin/etcd (TestRobustnessExploratoryKubernetesHighTrafficClusterOfSize3-test-0) (40381): panic: execute job failed
```
### What did you expect to happen?

Etcd not to panic

### How can we reproduce it (as minimally and precisely as possible)?

TODO

### Anything else we need to know?

_No response_

### Etcd version (please run commands below)

v3.6

### Etcd configuration (command line flags or environment variables)

# paste your configuration here

### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

```console
$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints= endpoint status -w table
# paste output here
```

### Relevant log output

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.