etcd-io / etcd-io/etcd

Panic in `IsLocalMemberLearner` when `Maintenance/Status` is called concurrently with member removal

Open
#21,966 2 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
3d 3h
Merged PRs (30d)
42

Description

### Bug report criteria

- [x] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [x] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [x] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.

### What happened?

`RaftCluster.IsLocalMemberLearner` panics with `"failed to find local ID in cluster members"` when a `Maintenance/Status` gRPC call is processed concurrently with a `ConfChangeRemoveNode` being applied on the same member. The result is an unclean shutdown (exit code 2) of the removed member instead of the expected clean exit (exit code 0).

The race requires a `Maintenance/Status` gRPC call to be executing inside the server-side handler on the victim member at the moment `ConfChangeRemoveNode` is applied. Using a persistent gRPC connection with concurrent callers reliably hits it.

A self-contained Go reproducer is attached below. It starts a 3-member cluster, waits for stability, then removes `m3` via `m1` while flooding `Maintenance/Status` calls from 64 concurrent goroutines on a pre-established connection to `m3`.

`m3` exits with code 2 (Go panic) instead of code 0.

### What did you expect to happen?

The removed member should exit cleanly with code 0, as it does when no concurrent `Maintenance/Status` calls are in flight.

### How can we reproduce it (as minimally and precisely as possible)?

reproduce_etcd_panic.go

```go
// reproduce_etcd_panic.go: etcd IsLocalMemberLearner() panic reproducer
//
// Starts a 3-member cluster, then removes m3 via m1 while flooding concurrent
// Maintenance/Status calls on a persistent gRPC connection to m3.
// Some goroutines are inside the handler when the Raft apply deletes
// localID from c.members -> IsLocalMemberLearner() panics -> exit code 2.
package main

import (
"bytes"
"context"
"fmt"
"log"
"os"
"os/exec"
"path/filepath"
"sync/atomic"
"time"

pb "go.etcd.io/etcd/api/v3/etcdserverpb"
clientv3 "go.etcd.io/etcd/client/v3"
"go.uber.org/zap"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)

const (
concurrency = 64
stabilityWait = 7 * time.Second
initialCluster = "m1=http://127.0.0.1:12380,m2=http://127.0.0.1:22380,m3=http://127.0.0.1:32380"
)

func main() {
dir, _ := os.MkdirTemp("", "")
defer os.RemoveAll(dir)
log.Printf("workdir: %s", dir)

log.Println("Starting cluster (m1, m2, m3)...")
startEtcd("m1", dir, "127.0.0.1:12379", "127.0.0.1:12380")
startEtcd("m2", dir, "127.0.0.1:22379", "127.0.0.1:22380")
m3 := startEtcd("m3", dir, "127.0.0.1:32379", "127.0.0.1:32380")

mgmt, _ := clientv3.New(clientv3.Config{
Endpoints: []string{"http://127.0.0.1:12379"},
DialTimeout: 15 * time.Second,
Logger: zap.NewNop(),
})
defer mgmt.Close()

log.Print("Waiting for cluster health...")
for {
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
_, err := mgmt.MemberList(ctx)
cancel()
if err == nil {
break
}
time.Sleep(500 * time.Millisecond)
}
log.Println("Cluster OK")

log.Printf("Waiting %s for peer connections to stabilize...", stabilityWait)
time.Sleep(stabilityWait)

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
list, _ := mgmt.MemberList(ctx)
cancel()
var m3id uint64
for _, m := range list.Members {
if m.Name == "m3" {
m3id = m.ID
}
}
log.Printf("m3 member ID: %x", m3id)

conn, _ := grpc.NewClient("127.0.0.1:32379",
grpc.WithTransportCredentials(insecure.NewCredentials()))
defer conn.Close()
mc := pb.NewMaintenanceClient(conn)

wctx, wcancel := context.WithTimeout(context.Background(), 5*time.Second)
mc.Status(wctx, &pb.StatusRequest{})
wcancel()
log.Println("Persistent gRPC connection to m3 established and warmed up")

var calls int64
stop := make(chan struct{})
for range concurrency {
go func() {
for {
select {
case <-stop:
return
default:
}
ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
mc.Status(ctx, &pb.StatusRequest{})
cancel()
atomic.AddInt64(&calls, 1)
}
}()
}
time.Sleep(50 * time.Millisecond)

log.Println("Hammer running. Removing m3 via m1...")
go func() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
mgmt.MemberRemove(ctx, m3id)
}()

time.Sleep(500 * time.Millisecond)
close(stop)

exitCode := -1
done := make(chan int, 1)
go func() {
if err := m3.Wait(); err == nil {
done <- 0
} else if e, ok := err.(*exec.ExitError); ok {
done <- e.ExitCode()
}
}()
select {
case exitCode = <-done:
case <-time.After(3 * time.Second):
}

log.Printf("m3 exit %d (%d Status calls)", exitCode, atomic.LoadInt64(&calls))

if exitCode == 2 {
fmt.Println("PANIC REPRODUCED!")
if b, _ := os.ReadFile(filepath.Join(dir, "m3.log")); b != nil {
const needle = "failed to find local ID"
if i := bytes.Index(b, []byte(needle)); i >= 0 {
start := bytes.LastIndex(b[:i], []byte("\n")) + 1
if j := bytes.Index(b[i+len(needle):], []byte(needle)); j >= 0 {
end := bytes.LastIndex(b[:i+len(needle)+j], []byte("\n")) + 1
b = b[start:end]
} else {
b = b[start:]
}
fmt.Printf("%s", b)
}
}
os.Exit(0)
}
fmt.Println("not triggered - try again")
os.Exit(1)
}

func startEtcd(name, dir, client, peer string) *exec.Cmd {
f, _ := os.Create(filepath.Join(dir, name+".log"))
cmd := exec.Command("etcd",
"--name", name,
"--data-dir", filepath.Join(dir, name),
"--listen-client-urls", "http://"+client,
"--advertise-client-urls", "http://"+client,
"--listen-peer-urls", "http://"+peer,
"--initial-advertise-peer-urls", "http://"+peer,
"--initial-cluster", initialCluster,
"--initial-cluster-state", "new",
"--initial-cluster-token", "repro",
"--log-format", "console",
)
cmd.Stdout, cmd.Stderr = f, f
_ = cmd.Start()
return cmd
}
```

Dockerfile

```dockerfile
# Dockerfile.reproduce: etcd IsLocalMemberLearner() panic reproducer
#
# Builds etcd from the main branch and runs the reproducer against it.
#
# Build: docker build -t etcd-panic-repro .
# Run: docker run --rm -ti etcd-panic-repro

FROM golang:1.26.4-alpine

RUN apk add --no-cache git

# Clone etcd main branch
RUN git clone --depth=1 https://github.com/etcd-io/etcd.git /etcd

# Build the etcd server binary; the repo's own go.work handles module resolution
RUN cd /etcd && go build -o /usr/local/bin/etcd ./server

# Build the reproducer using a Go workspace that pulls in the local etcd
# sub-modules, avoiding the need for a tagged release version
WORKDIR /src
COPY reproduce_etcd_panic.go .
RUN go mod init repro && \
go work init . && \
find /etcd -mindepth 1 -maxdepth 3 -name go.mod \
-not -path '*/vendor/*' \
-exec dirname {} \; | xargs go work use && \
go build -o /reproduce .

CMD ["/reproduce"]
```

### Anything else we need to know?

### Analysis

The race involves two concurrent goroutines inside the removed member:

1. **Raft apply goroutine**: processes `ConfChangeRemoveNode`, calls `RaftCluster.RemoveMember()` which immediately deletes `localID` from `c.members`, then schedules shutdown.
2. **gRPC handler goroutine**: processes an in-flight `Maintenance/Status` request. `maintenanceServer.Status()` calls `s.IsLearner()` which calls `IsLocalMemberLearner()`.

During the interval between the `ConfChange` being applied and closing the gRPC server, any concurrent `Status` call reaches this code in `membership/cluster.go`:

```go
func (c *RaftCluster) IsLocalMemberLearner() bool {
c.Lock()
defer c.Unlock()
localMember, ok := c.members[c.localID]
if !ok {
c.lg.Panic( // <-- panics instead of returning false
"failed to find local ID in cluster members",
...
)
}
return localMember.IsLearner
}
```

Because `c.members[localID]` has already been deleted by the apply goroutine, `ok` is `false` and the panic fires.

### Suggested fix

`IsLocalMemberLearner` is only used to determine whether the local member is a Raft learner. Returning `false` when the local ID is not found is correct and should be safe, a member that has been removed is not a learner. Replacing the `Panic` with a `Warn` and returning `false` eliminates the crash:

```go
func (c *RaftCluster) IsLocalMemberLearner() bool {
c.Lock()
defer c.Unlock()
localMember, ok := c.members[c.localID]
if !ok {
c.lg.Warn(
"local member ID not found in cluster members; member may have been removed",
zap.String("local-member-id", c.localID.String()),
)
return false
}
return localMember.IsLearner
}
```

### Etcd version (please run commands below)

main branch

### Etcd configuration (command line flags or environment variables)

see Dockerfile

### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

_No response_

### Relevant log output

```Shell
2026-06-15T14:06:52.359956Z panic membership/cluster.go:889 failed to find local ID in cluster members {"cluster-id": "adfa0f9fbbd0aa45", "local-member-id": "c0037e87680c575"}
go.etcd.io/etcd/server/v3/etcdserver/api/membership.(*RaftCluster).IsLocalMemberLearner
go.etcd.io/etcd/server/v3/etcdserver/api/membership/cluster.go:889
go.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).IsLearner
go.etcd.io/etcd/server/v3/etcdserver/server.go:2566
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*maintenanceServer).Status
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc/maintenance.go:270
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*authMaintenanceServer).Status
go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc/maintenance.go:371
go.etcd.io/etcd/api/v3/etcdserverpb._Maintenance_Status_Handler.func1
go.etcd.io/etcd/api/v3@v3.6.12/etcdserverpb/rpc.pb.go:7691
...
google.golang.org/grpc.(*Server).serveStreams.func2.1
google.golang.org/grpc@v1.79.3/server.go:1064
```

Contributor guide

Open the contributing guide

Research direction

Start with IsLocalMemberLearner in server/etcdserver/api/membership/cluster.go and the Status handler in server/etcdserver/api/v3rpc/maintenance.go. Run the attached Dockerfile.reproduce.go reproducer against the reported concurrent member removal scenario, then verify that the removed member exits cleanly without the reported panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
api, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.