leasing: LockWriteOps range branch races with lease-cache mutation
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
### Bug report criteria
- [x] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [x] I have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [x] Existing open issues and the etcd [frequently asked questions](https://etcd.io/docs/latest/faq/) have been checked and this is not a duplicate.
### What happened?
`leaseCache.LockWriteOps` handles single-key writes through `lc.Lock`, which
holds `lc.mu`. Its range branch instead iterates `lc.entries` without holding
that mutex:
```go
for k := range lc.entries {
...
if wc, _ := lc.Lock(k); wc != nil {
ret = append(ret, wc)
}
}
```
Other cache operations such as `Add` and `Evict` mutate `lc.entries` while
holding `lc.mu`. The race detector reports the unsynchronized map iteration
against those writes. Outside the race detector, concurrent map iteration and
mutation can also terminate the process.
### What did you expect to happen?
The range branch should synchronize access to `lc.entries` just like the other
cache operations. Running the focused test with `-race` should not report a data
race.
### How can we reproduce it (as minimally and precisely as possible)?
Add this complete unit test as
`client/v3/leasing/cache_lock_write_ops_test.go` and run:
```console
cd client/v3
go test -race ./leasing -run '^TestLeaseCacheLockWriteOpsRangeSynchronizesEntries$' -count=1
```
```go
// Copyright 2026 The etcd Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package leasing
import (
"context"
"fmt"
"testing"
"time"
"go.etcd.io/etcd/api/v3/etcdserverpb"
"go.etcd.io/etcd/api/v3/mvccpb"
clientv3 "go.etcd.io/etcd/client/v3"
)
func TestLeaseCacheLockWriteOpsRangeSynchronizesEntries(t *testing.T) {
lc := leaseCache{
entries: map[string]*leaseKey{
"k/initial": leaseKeyForLockWriteOpsRange("k/initial", 1),
},
revokes: make(map[string]time.Time),
header: &etcdserverpb.ResponseHeader{Revision: 1},
}
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
donec := make(chan struct{}, 2)
go func() {
defer func() { donec <- struct{}{} }()
for ctx.Err() == nil {
wcs := lc.LockWriteOps([]clientv3.Op{
clientv3.OpDelete("k/", clientv3.WithRange("k0")),
})
closeAll(wcs)
}
}()
go func() {
defer func() { donec <- struct{}{} }()
for i := 0; ctx.Err() == nil; i++ {
key := fmt.Sprintf("k/%d", i)
lc.Add(key, getResponseForLockWriteOpsRange(key, int64(i+2)), clientv3.OpGet(key))
if i%2 == 0 {
lc.Evict(key)
}
}
}()
<-donec
<-donec
}
func leaseKeyForLockWriteOpsRange(key string, rev int64) *leaseKey {
return &leaseKey{
response: getResponseForLockWriteOpsRange(key, rev),
rev: rev,
}
}
func getResponseForLockWriteOpsRange(key string, rev int64) *clientv3.GetResponse {
return &clientv3.GetResponse{
Header: &etcdserverpb.ResponseHeader{Revision: rev},
Kvs: []*mvccpb.KeyValue{
{
Key: []byte(key),
Value: []byte("value"),
CreateRevision: rev,
ModRevision: rev,
Version: 1,
},
},
Count: 1,
}
}
```
### Anything else we need to know?
The unsynchronized iteration is in the [current `LockWriteOps` range
branch](https://github.com/etcd-io/etcd/blob/12caed621b38312cc1084113415bf135c59e6457/client/v3/leasing/cache.go#L85-L107).
The existing `leaseCache.LockRange` already performs the range scan and replaces
the matching wait channels while holding `lc.mu`; the range branch can reuse or
factor that synchronization. A fix should avoid taking `lc.mu` and then calling
`lc.Lock`, because `lc.Lock` acquires the same mutex.
### Etcd version (please run commands below)
Confirmed on:
```console
$ git rev-parse HEAD
12caed621b38312cc1084113415bf135c59e6457
$ go version
go version go1.26.5 linux/amd64
the same unsynchronized range loop is present at v3.6.13
(b0f9ef190952e6e66a778513097a02ee41220727)
```
### Etcd configuration (command line flags or environment variables)
No server configuration is needed; this is a client-side unit test.
### Etcd debug information
Not applicable.
### Relevant log output
```text
WARNING: DATA RACE
Read at ... by goroutine ...:
go.etcd.io/etcd/client/v3/leasing.(*leaseCache).LockWriteOps()
client/v3/leasing/cache.go:96
Previous write at ... by goroutine ...:
go.etcd.io/etcd/client/v3/leasing.(*leaseCache).Add()
client/v3/leasing/cache.go:133
FAIL go.etcd.io/etcd/client/v3/leasing
```
Contributor guide
Research direction
Start with client/v3/leasing/cache.go, especially the LockWriteOps range branch and the existing LockRange implementation. Add the supplied test in client/v3/leasing/cache_lock_write_ops_test.go, then run go test -race ./leasing -run '^TestLeaseCacheLockWriteOpsRangeSynchronizesEntries$' -count=1. Done means the focused test passes without a data-race report.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100