tools/benchmark: watch event latency measures client-side batch processing rather than end-to-end delivery
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
### Bug report criteria
- [x] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [ ] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [ ] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.
### What happened?
The `Watch events received summary` latency generated by
`tools/benchmark/cmd/watch.go` starts timing only after a `WatchResponse` has
already been received. It then uses one start timestamp for every event in that
response batch.
As a result, the reported latency measures client-side iteration and report-channel
submission after delivery, and is strongly affected by response batch size and
goroutine scheduling. It does not measure Put-to-Watch end-to-end delivery
latency.
I observed this with:
```text
etcd commit e838ef116fc368b321ddf2e424167b15174fb80d
version: bump up to 3.6.7
```
I checked `main` on 2026-07-30 and the same implementation is still present.
The benchmark was run against a healthy local three-member cluster:
```bash
benchmark + --endpoints=http://127.0.0.1:12379,http://127.0.0.1:22379,http://127.0.0.1:32379 + --clients=10 --conns=10 + watch + --key-size=32 + --streams=10 + --watch-per-stream=100 + --watched-key-total=1000 + --key-space-size=1 + --put-rate=500 + --put-total=36000
```
The cluster health check succeeded before each trial. Each trial rebuilt the
three-member cluster and completed the expected fixed work.
Across independent trials, the reported Watch event P99 repeatedly formed fast
and slow groups. For example:
```text
trial set A P99 seconds:
0.0002, 0.0085, 0.0086, 0.0094, 0.0094
trial set B P99 seconds:
0.0002, 0.0003, 0.0003, 0.0096, 0.0104
```
The median changes from 8.6 ms to 0.3 ms when the number of fast and slow trials
changes from 2:3 to 3:2.
### What did you expect to happen?
I think one of the following would make the metric unambiguous:
1. Rename and document the existing measurement as client-side event-batch
processing/reporting latency, and report response batch sizes alongside it.
2. Add a separate end-to-end Watch delivery latency metric.
For an end-to-end metric, each Put could carry a unique sequence value. The
benchmark process could record a monotonic start timestamp before issuing that
Put, then match the sequence value when each watcher receives the event. This
would allow the benchmark to report a clearly defined request-start-to-event
delivery distribution. A commit-complete-to-delivery metric could also be added,
but would need to handle the race where a Watch event arrives before the Put
response is recorded.
It would also be useful to wait for server-side watch establishment before
starting the Put workload, so watch creation timing is not mixed with event
delivery.
#### Relevant code
`recvWatchChan` currently does:
```go
for r := range wch {
st := time.Now()
for range r.Events {
results <- report.Result{Start: st, End: time.Now()}
bar.Increment()
if atomic.AddInt32(nrRxed, -1) <= 0 {
return
}
}
}
```
This has three consequences:
1. `st` is recorded after the `WatchResponse` reaches the client.
2. All events in one response share the same `st`, so later events include more
loop and reporting overhead.
3. Sending to `results` can include local channel backpressure and scheduler
delay in the measured value.
The output therefore describes local response-batch processing latency, not event
delivery latency.
### How can we reproduce it (as minimally and precisely as possible)?
```bash
benchmark + --endpoints=http://127.0.0.1:12379,http://127.0.0.1:22379,http://127.0.0.1:32379 + --clients=10 --conns=10 + watch + --key-size=32 + --streams=10 + --watch-per-stream=100 + --watched-key-total=1000 + --key-space-size=1 + --put-rate=500 + --put-total=36000
```
### Anything else we need to know?
_No response_
### Etcd version (please run commands below)
```console
$ etcd --version
etcd Version: 3.6.7
Git SHA: e838ef1
Go Version: go1.24.11
Go OS/Arch: linux/amd64
$ etcdctl version
etcdctl version: 3.6.7
API version: 3.6
```
### Etcd configuration (command line flags or environment variables)
# paste your configuration here
### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
```console
$ etcdctl member list -w table
# paste output here
$ etcdctl --endpoints= endpoint status -w table
# paste output here
```
### Relevant log output
```Shell
```
Contributor guide
Research direction
Start in tools/benchmark/cmd/watch.go, especially recvWatchChan, and reproduce the behavior with the benchmark command in the issue. Trace how WatchResponse timestamps and results are recorded, then define whether the work will rename the existing client-side metric or add end-to-end delivery latency. Done means the reported metric has an unambiguous documented meaning and no longer presents batch-processing latency as delivery latency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100