etcd-io / etcd-io/etcd

Maintenance Status RPC can SIGSEGV a member during shutdown: StorageVersion reads from a closed backend

Open
#22,189 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
3d 1h
Merged PRs (30d)
44

Description

### Bug report criteria

- [x] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [x] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [x] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.

### What happened?

A `Maintenance/Status` gRPC call processed while the member is shutting down (e.g. after a `ConfChangeRemoveNode` removing that member) can crash the process with a SIGSEGV instead of letting it exit cleanly.

The shutdown sequence in the run goroutine's defer (`server/etcdserver/server.go`) closes `s.stopping`, waits for attached goroutines, and then calls `Cleanup()`, which closes the backend (`s.be.Close()`) without any synchronization against in-flight readers. A concurrent `Status` handler that reaches `EtcdServer.StorageVersion()` in that window calls `schema.DetectSchemaVersion()` on the closed backend. The backend's final commit (`CommitAndStop`) has already reset the shared read transaction to `nil`, so `backend.(*baseReadTx).UnsafeRange()` calls `(*bolt.Tx).Bucket()` on a nil `*bolt.Tx` and the process dies:

```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x104c3eb1c]

goroutine 6351 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
go.etcd.io/bbolt@v1.5.0/tx.go:112
go.etcd.io/etcd/server/v3/storage/backend.(*baseReadTx).UnsafeRange(...)
server/storage/backend/read_tx.go:103 +0x1ec
go.etcd.io/etcd/server/v3/storage/schema.UnsafeReadStorageVersion(...)
server/storage/schema/version.go:35 +0x60
go.etcd.io/etcd/server/v3/storage/schema.UnsafeDetectSchemaVersion(...)
server/storage/schema/schema.go:93 +0x40
go.etcd.io/etcd/server/v3/storage/schema.DetectSchemaVersion(...)
server/storage/schema/schema.go:88 +0xac
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).StorageVersion(...)
server/etcdserver/server.go:2193 +0xe4
go.etcd.io/etcd/server/v3/etcdserver.(*serverVersionAdapter).GetStorageVersion(...)
server/etcdserver/adapters.go:73 +0x20
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*maintenanceServer).Status(...)
server/etcdserver/api/v3rpc/maintenance.go:280 +0x210
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*authMaintenanceServer).Status(...)
server/etcdserver/api/v3rpc/maintenance.go:374 +0x68
go.etcd.io/etcd/api/v3/etcdserverpb._Maintenance_Status_Handler.func1(...)
api/etcdserverpb/rpc_grpc.pb.go:1243
... (grpc server internals)
```

Note on discoverability: on current `main` this crash is masked in the `Status` path because the same handler panics earlier in `RaftCluster.IsLocalMemberLearner` (#21966). With that panic fixed (#22188), the reproducer for #21966 immediately surfaces this second, independent crash in the same shutdown window.

### What did you expect to happen?

The removed/stopping member exits cleanly with code 0. `Status` calls racing with shutdown should either succeed or fail with a gRPC error, never crash the server.

### How can we reproduce it (as minimally and precisely as possible)?

Use the reproducer program from #21966 unchanged (3-member cluster, 64 goroutines hammering `Maintenance/Status` on a persistent gRPC connection to m3, then removing m3 via m1), against a build that includes the #22188 fix (otherwise the `IsLocalMemberLearner` panic fires first and masks this one):

1. Build etcd from `main` + #22188 (`go build -o bin/etcd ./server`).
2. Run the reproducer from #21966 with that binary on `PATH`.
3. m3 exits with code 2 and the SIGSEGV stack above in `m3.log` (reproduced on the first attempt in my runs).

With `StorageVersion` prevented from reading a closed backend, the same reproducer gives 3/3 clean exits with code 0 (~300k Status calls per run).

### Anything else we need to know?

- Root cause is the unsynchronized ordering in the run goroutine's defer: `close(s.stopping)` → `s.wg.Wait()` → `s.Cleanup()` (which calls `s.be.Close()`), while gRPC handlers may still be executing. `StorageVersion()` takes `s.bemu.RLock()`, but `Cleanup()` does not take `bemu` before closing the backend, so the lock does not protect readers from the close.
- Internal callers of `StorageVersion` (e.g. `monitorStorageVersion`) are safe: they run via `GoAttach` and are waited on by `s.wg.Wait()` before `Cleanup()` runs. Only externally driven paths (the `Status` RPC via `serverVersionAdapter.GetStorageVersion`) race.
- The other backend accesses in the `Status` handler (`Backend().Size()`, `SizeInUse()`) read atomics and do not crash.
- A fix is prepared: close the backend under `s.bemu` in `Cleanup()` and check `s.stopping` under the `bemu` read lock in `StorageVersion()`, returning nil (the handler already omits the field for a nil version).

### Etcd version (please run commands below)

```console
$ etcd --version
etcd Version: 3.8.0-alpha.0
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.26.5
Go OS/Arch: darwin/arm64
```

Built from `main` (14bd0caac) plus #22188. The affected code (`Cleanup`/`StorageVersion` ordering) is unchanged on `main`; the same pattern exists on the release-3.6 branch (`StorageVersion` reads under `bemu.RLock` while `Cleanup` closes the backend without taking `bemu`), though there the Status path is shadowed by the #21966 panic. release-3.5 is unaffected: it predates the `StorageVersion`/schema machinery entirely.

### Etcd configuration (command line flags or environment variables)

Defaults as set by the #21966 reproducer (3 members, plain HTTP, `--log-format console`); no non-default tuning is required.

Contributor guide

Open the contributing guide

Research direction

Start in server/etcdserver/server.go at Cleanup() and StorageVersion(), then inspect the Maintenance Status path in server/etcdserver/api/v3rpc/maintenance.go. Build ./server and run the reproducer from issue #21966 against a build including #22188. Done means concurrent Status calls during member removal never panic and the removed member exits cleanly with code 0.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
api, backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.