etcd-io / etcd-io/etcd

Deflake TestEtcdHealthyWithTinySnapshotCatchupEntries

Open
#21,852 6 comments 0 reactions 1 assignee Claimed by @henrybear327 View on GitHub
area/testing stale type/flake
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
3d 1h
Merged PRs (30d)
44

Description

### Which Github Action / Prow Jobs are flaking?

ci-etcd-e2e-arm64

### Which tests are flaking?

TestEtcdHealthyWithTinySnapshotCatchupEntries

See
- https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-e2e-arm64/2059701081549574144
- https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-e2e-arm64/2059761480345063424
- https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-e2e-arm64/2060064227866972160
- https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-etcd-e2e-arm64/2060245927020466176

### Github Action / Prow Job link

_No response_

### Reason for failure (if possible)

In all the cases linked above...

We can see the PUT timed out
```
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl (/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl_--endpoints=http://localhost:20000,http://localhost:20005,http://localhost:20010_put_foo_bar7_-w_json) (29611): {"level":"warn","ts":"2026-05-27T22:31:44.337908Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x8ccde8cba00/localhost:20000","peer":"Peer{Addr: '127.0.0.1:20000', LocalAddr: '127.0.0.1:59864', AuthInfo: 'insecure'}","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL"}
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl (/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl_--endpoints=http://localhost:20000,http://localhost:20005,http://localhost:20010_put_foo_bar7_-w_json) (29611): Error: context deadline exceeded
etcd_config_test.go:631:
Error: Received unexpected error:
match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/home/prow/go/src/github.com/etcd-io/etcd/bin/etcdctl --endpoints=http://localhost:20000/,http://localhost:20005/,http://localhost:20010/ put foo bar7 -w json]]
```

We can also see lots of put request taking almost 5s.
```
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd (TestEtcdHealthyWithTinySnapshotCatchupEntries-test-0) (26967): {"level":"warn","ts":"2026-05-27T22:31:44.336805Z","caller":"v3rpc/interceptor.go:203","msg":"request stats","start time":"2026-05-27T22:31:39.339947Z","time spent":"4.996775752s","remote":"127.0.0.1:59862","response type":"/etcdserverpb.KV/Put","request count":1,"request size":11,"response count":0,"response size":0,"request content":"key:\"foo\" value_size:4 "}
```

We can also see `sent merged snapshot` taking >5s
```
/home/prow/go/src/github.com/etcd-io/etcd/bin/etcd (TestEtcdHealthyWithTinySnapshotCatchupEntries-test-1) (26968): {"level":"info","ts":"2026-05-27T22:31:45.458787Z","caller":"etcdserver/server.go:1880","msg":"sent merged snapshot","from":"5ac8aa22f1eb4c8f","to":"ca50e9357181d758","bytes":87131,"size":"87 kB","took":"6.11400122s"}
```

### Anything else we need to know?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.