Deflake TestEtcdHealthyWithTinySnapshotCatchupEntries
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 44
Description
### Which Github Action / Prow Jobs are flaking?
ci-etcd-e2e-arm64
### Which tests are flaking?
TestEtcdHealthyWithTinySnapshotCatchupEntries
See
- https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-e2e-arm64/2059701081549574144
- https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-e2e-arm64/2059761480345063424
- https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-e2e-arm64/2060064227866972160
- https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-e2e-arm64/2060245927020466176
### Github Action / Prow Job link
_No response_
### Reason for failure (if possible)
In all the cases linked above...
We can see the PUT timed out
```
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl (/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl_--endpoints=http://localhost:20000,http://localhost:20005,http://localhost:20010_put_foo_bar7_-w_json) (29611): {"level":"warn","ts":"2026-05-27T22:31:44.337908Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x8ccde8cba00/localhost:20000","peer":"Peer{Addr: '127.0.0.1:20000', LocalAddr: '127.0.0.1:59864', AuthInfo: 'insecure'}","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL"}
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl (/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl_--endpoints=http://localhost:20000,http://localhost:20005,http://localhost:20010_put_foo_bar7_-w_json) (29611): Error: context deadline exceeded
etcd_config_test.go:631:
Error: Received unexpected error:
match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl --endpoints=http://localhost:20000/,http://localhost:20005/,http://localhost:20010/ put foo bar7 -w json]]
```
We can also see lots of put request taking almost 5s.
```
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd (TestEtcdHealthyWithTinySnapshotCatchupEntries-test-0) (26967): {"level":"warn","ts":"2026-05-27T22:31:44.336805Z","caller":"v3rpc/interceptor.go:203","msg":"request stats","start time":"2026-05-27T22:31:39.339947Z","time spent":"4.996775752s","remote":"127.0.0.1:59862","response type":"/etcdserverpb.KV/Put","request count":1,"request size":11,"response count":0,"response size":0,"request content":"key:\"foo\" value_size:4 "}
```
We can also see `sent merged snapshot` taking >5s
```
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd (TestEtcdHealthyWithTinySnapshotCatchupEntries-test-1) (26968): {"level":"info","ts":"2026-05-27T22:31:45.458787Z","caller":"etcdserver/server.go:1880","msg":"sent merged snapshot","from":"5ac8aa22f1eb4c8f","to":"ca50e9357181d758","bytes":87131,"size":"87 kB","took":"6.11400122s"}
```
### Anything else we need to know?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.