Unexpected Unknown grpc code for ErrNotPrimary
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
### Bug report criteria
- [x] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [x] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [x] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.
### What happened?
Due to a performance hiccup there was a leader switch, which apparently caused `/v3/lease/keepalive` requests return an error: `{"code": 2, "message": "not a primary lessor"}`. Everything is as expected, except the Unknown error code.
### What did you expect to happen?
Some error code that signifies that retry is in order. Unavailable maybe.
### How can we reproduce it (as minimally and precisely as possible)?
Have `s.ensureLeadership()` check fail in `EtcdServer.LeaseRenew()`.
### Anything else we need to know?
_No response_
### Etcd version (please run commands below)
3.6.7
### Etcd configuration (command line flags or environment variables)
_No response_
### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
_No response_
### Relevant log output
```Shell
{"level":"warn","ts":"2026-04-21T22:04:25.607018+0200","caller":"etcdserver/raft.go:387","msg":"leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk","to":"95012eee2c61378b","heartbeat-interval":"250ms","expected-duration":"500ms","exceeded-duration":"1.484060419s"}
{"level":"warn","ts":"2026-04-21T22:04:25.607340+0200","caller":"etcdserver/raft.go:387","msg":"leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk","to":"836bb8986617b0d6","heartbeat-interval":"250ms","expected-duration":"500ms","exceeded-duration":"1.484390506s"}
{"level":"warn","ts":"2026-04-21T22:04:25.742549+0200","caller":"etcdserver/v3_server.go:911","msg":"waiting for ReadIndex response took too long, retrying","sent-request-id":10377592809537642783,"retry-timeout":"500ms"}
{"level":"info","ts":"2026-04-21T22:04:26.006035+0200","logger":"raft","caller":"v3@v3.6.0/raft.go:1101","msg":"a77ddc750103e129 [logterm: 216, index: 26712135, vote: a77ddc750103e129] ignored MsgPreVote from 95012eee2c61378b [logterm: 216, index: 26712130] at term 216: lease is not expired (remaining ticks: 1)"}
{"level":"info","ts":"2026-04-21T22:04:26.006102+0200","logger":"raft","caller":"v3@v3.6.0/raft.go:1101","msg":"a77ddc750103e129 [logterm: 216, index: 26712135, vote: a77ddc750103e129] ignored MsgVote from 95012eee2c61378b [logterm: 216, index: 26712130] at term 216: lease is not expired (remaining ticks: 1)"}
{"level":"info","ts":"2026-04-21T22:04:26.006137+0200","logger":"raft","caller":"v3@v3.6.0/raft.go:1116","msg":"a77ddc750103e129 [term: 216] received a MsgHeartbeat message with higher term from 95012eee2c61378b [term: 217]"}
{"level":"info","ts":"2026-04-21T22:04:26.006207+0200","logger":"raft","caller":"v3@v3.6.0/raft.go:897","msg":"a77ddc750103e129 became follower at term 217"}
{"level":"info","ts":"2026-04-21T22:04:26.006238+0200","logger":"raft","caller":"v3@v3.6.0/node.go:372","msg":"raft.node: a77ddc750103e129 changed leader from a77ddc750103e129 to 95012eee2c61378b at term 217"}
{"level":"warn","ts":"2026-04-21T22:04:26.006316+0200","caller":"etcdserver/server.go:939","msg":"Failed to check current member's leadership","error":"etcdserver: leader changed"}
{"level":"info","ts":"2026-04-21T22:04:26.008988+0200","logger":"raft","caller":"v3@v3.6.0/log.go:158","msg":"found conflict at index 26712131 [existing term: 216, conflicting term: 217]"}
{"level":"info","ts":"2026-04-21T22:04:26.009030+0200","logger":"raft","caller":"v3@v3.6.0/log_unstable.go:203","msg":"replace the unstable entries from index 26712131"}
{"level":"info","ts":"2026-04-21T22:04:26.019329+0200","caller":"etcdserver/v3_server.go:903","msg":"first commit in current term: resending ReadIndex request"}
{"level":"warn","ts":"2026-04-21T22:04:26.039277+0200","caller":"etcdserver/server.go:946","msg":"Current member isn't a leader","local-member-id":12069044971509113129,"new-lead":10736914587109635979}
{"level":"warn","ts":"2026-04-21T22:04:26.039397+0200","caller":"etcdserver/server.go:946","msg":"Current member isn't a leader","local-member-id":12069044971509113129,"new-lead":10736914587109635979}
...
```
Contributor guide
Research direction
Start at EtcdServer.LeaseRenew and the ensureLeadership check, then trace how its ErrNotPrimary result is converted for /v3/lease/keepalive. Reproduce the failed leadership check described in the issue and inspect the gRPC error mapping. Done means the response uses an error code that signals clients should retry instead of Unknown, with coverage for this path if the surrounding tests support it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grpc
- Domain
- api, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100