Maintenance Status RPC can SIGSEGV a member during shutdown: StorageVersion reads from a closed backend
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 44
Description
### Bug report criteria
- [x] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [x] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [x] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.
### What happened?
A `Maintenance/Status` gRPC call processed while the member is shutting down (e.g. after a `ConfChangeRemoveNode` removing that member) can crash the process with a SIGSEGV instead of letting it exit cleanly.
The shutdown sequence in the run goroutine's defer (`server/etcdserver/server.go`) closes `s.stopping`, waits for attached goroutines, and then calls `Cleanup()`, which closes the backend (`s.be.Close()`) without any synchronization against in-flight readers. A concurrent `Status` handler that reaches `EtcdServer.StorageVersion()` in that window calls `schema.DetectSchemaVersion()` on the closed backend. The backend's final commit (`CommitAndStop`) has already reset the shared read transaction to `nil`, so `backend.(*baseReadTx).UnsafeRange()` calls `(*bolt.Tx).Bucket()` on a nil `*bolt.Tx` and the process dies:
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x104c3eb1c]
goroutine 6351 [running]:
go.etcd.io/bbolt.(*Tx).Bucket(...)
go.etcd.io/bbolt@v1.5.0/tx.go:112
go.etcd.io/etcd/server/v3/storage/backend.(*baseReadTx).UnsafeRange(...)
server/storage/backend/read_tx.go:103 +0x1ec
go.etcd.io/etcd/server/v3/storage/schema.UnsafeReadStorageVersion(...)
server/storage/schema/version.go:35 +0x60
go.etcd.io/etcd/server/v3/storage/schema.UnsafeDetectSchemaVersion(...)
server/storage/schema/schema.go:93 +0x40
go.etcd.io/etcd/server/v3/storage/schema.DetectSchemaVersion(...)
server/storage/schema/schema.go:88 +0xac
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).StorageVersion(...)
server/etcdserver/server.go:2193 +0xe4
go.etcd.io/etcd/server/v3/etcdserver.(*serverVersionAdapter).GetStorageVersion(...)
server/etcdserver/adapters.go:73 +0x20
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*maintenanceServer).Status(...)
server/etcdserver/api/v3rpc/maintenance.go:280 +0x210
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*authMaintenanceServer).Status(...)
server/etcdserver/api/v3rpc/maintenance.go:374 +0x68
go.etcd.io/etcd/api/v3/etcdserverpb._Maintenance_Status_Handler.func1(...)
api/etcdserverpb/rpc_grpc.pb.go:1243
... (grpc server internals)
```
Note on discoverability: on current `main` this crash is masked in the `Status` path because the same handler panics earlier in `RaftCluster.IsLocalMemberLearner` (#21966). With that panic fixed (#22188), the reproducer for #21966 immediately surfaces this second, independent crash in the same shutdown window.
### What did you expect to happen?
The removed/stopping member exits cleanly with code 0. `Status` calls racing with shutdown should either succeed or fail with a gRPC error, never crash the server.
### How can we reproduce it (as minimally and precisely as possible)?
Use the reproducer program from #21966 unchanged (3-member cluster, 64 goroutines hammering `Maintenance/Status` on a persistent gRPC connection to m3, then removing m3 via m1), against a build that includes the #22188 fix (otherwise the `IsLocalMemberLearner` panic fires first and masks this one):
1. Build etcd from `main` + #22188 (`go build -o bin/etcd ./server`).
2. Run the reproducer from #21966 with that binary on `PATH`.
3. m3 exits with code 2 and the SIGSEGV stack above in `m3.log` (reproduced on the first attempt in my runs).
With `StorageVersion` prevented from reading a closed backend, the same reproducer gives 3/3 clean exits with code 0 (~300k Status calls per run).
### Anything else we need to know?
- Root cause is the unsynchronized ordering in the run goroutine's defer: `close(s.stopping)` → `s.wg.Wait()` → `s.Cleanup()` (which calls `s.be.Close()`), while gRPC handlers may still be executing. `StorageVersion()` takes `s.bemu.RLock()`, but `Cleanup()` does not take `bemu` before closing the backend, so the lock does not protect readers from the close.
- Internal callers of `StorageVersion` (e.g. `monitorStorageVersion`) are safe: they run via `GoAttach` and are waited on by `s.wg.Wait()` before `Cleanup()` runs. Only externally driven paths (the `Status` RPC via `serverVersionAdapter.GetStorageVersion`) race.
- The other backend accesses in the `Status` handler (`Backend().Size()`, `SizeInUse()`) read atomics and do not crash.
- A fix is prepared: close the backend under `s.bemu` in `Cleanup()` and check `s.stopping` under the `bemu` read lock in `StorageVersion()`, returning nil (the handler already omits the field for a nil version).
### Etcd version (please run commands below)
```console
$ etcd --version
etcd Version: 3.8.0-alpha.0
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.26.5
Go OS/Arch: darwin/arm64
```
Built from `main` (14bd0caac) plus #22188. The affected code (`Cleanup`/`StorageVersion` ordering) is unchanged on `main`; the same pattern exists on the release-3.6 branch (`StorageVersion` reads under `bemu.RLock` while `Cleanup` closes the backend without taking `bemu`), though there the Status path is shadowed by the #21966 panic. release-3.5 is unaffected: it predates the `StorageVersion`/schema machinery entirely.
### Etcd configuration (command line flags or environment variables)
Defaults as set by the #21966 reproducer (3 members, plain HTTP, `--log-format console`); no non-default tuning is required.
Contributor guide
Research direction
Start in server/etcdserver/server.go at Cleanup() and StorageVersion(), then inspect the Maintenance Status path in server/etcdserver/api/v3rpc/maintenance.go. Build ./server and run the reproducer from issue #21966 against a build including #22188. Done means concurrent Status calls during member removal never panic and the removed member exits cleanly with code 0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grpc
- Domain
- api, backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100