etcd-io / etcd-io/etcd

Missing snapshot-directory fsync can make a member fail after a crash

Open
#22,340 1 comment 0 reactions 1 assignee Claimed by @gyuho View on GitHub
type/bug
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
3d 1h
Merged PRs (30d)
44

Description

### Bug report criteria

- [x] This bug report is not security related, security issues should be disclosed privately via [the report form](https://github.com/etcd-io/etcd/security/advisories/new).
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [x] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [x] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.

### What happened?

When a member receives a database snapshot, `SaveDBFrom` syncs the snapshot file and renames it into `member/snap`, but does not sync the containing directory. After a power loss, the `*.snap.db` directory entry may be lost even though the file was synced.

The WAL can retain the corresponding snapshot record. On restart, etcd finds that record but not the database snapshot and panics:

`failed to find database snapshot file (snap: snapshot file doesn't exist)`

### What did you expect to happen?

etcd should make the renamed database snapshot durable before recording or processing the corresponding Raft snapshot. A directory-sync failure should stop snapshot handling and return an error.

### How can we reproduce it (as minimally and precisely as possible)?

1. Start a three-member cluster with aggressive snapshotting.
2. Stop or isolate one follower, write enough data to compact the Raft log, then restore the follower so it must receive a database snapshot.
3. Pause the follower after `*.snap.db` is renamed but before the snapshot directory is synced.
4. Kill etcd, then abruptly reboot the machine without syncing disks, for example with `echo b > /proc/sysrq-trigger`.
5. Restart the member. If the directory entry was lost but the WAL snapshot record survived, etcd panics because the referenced database snapshot is missing.

## Environment and affected version

- Three-member etcd cluster on Linux/amd64
- Reproduced on AWS EC2 with the member data directory on a dedicated non-journaled ext2 EBS volume
- Affected main revision: `23a4e406a2e70a807486b4c40a9e24da493886bf` (the base of PR #22314)
- Relevant configuration: aggressive snapshotting and one follower forced far enough behind to receive a full database snapshot

### Anything else we need to know?

## Similar issues

Issues [#11949](https://github.com/etcd-io/etcd/issues/11949) and [#14569](https://github.com/etcd-io/etcd/issues/14569) show the same startup failure but do not identify the missing directory fsync.

## Proposed fix and backport

PR [#22314](https://github.com/etcd-io/etcd/pull/22314) fixes this by syncing the snapshot directory on both the rename and existing-file paths, and propagating sync failures before Raft processing.

**We need to backport this fix to the v3.5, v3.6, and v3.7 release branches.**

### Etcd version (please run commands below)

Affected main revision: `23a4e406a2e70a807486b4c40a9e24da493886bf` (the base of PR #22314). The bug predates this revision; the same code path exists in the v3.5, v3.6, and v3.7 release branches.

```console
$ etcd --version
v3.7.1
```

### Etcd configuration (command line flags or environment variables)

Three-member cluster on Linux/amd64, reproduced on AWS EC2 with the member data directory on a dedicated non-journaled ext2 EBS volume. Aggressive snapshotting was used to force snapshot transfer; otherwise defaults.

### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Not applicable; the failure is at startup after a power loss, before the member can serve.

### Relevant log output

```Shell
panic: failed to recover v3 backend from snapshot: failed to find database snapshot file (snap: snapshot file doesn't exist)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.