After a large number of watch connections are disconnected from a client at the same time, the new watch cannot work properly.
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
### Bug report criteria
- [ ] This bug report is not security related, security issues should be disclosed privately via [etcd maintainers](mailto:etcd-maintainers@googlegroups.com).
- [ ] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [ ] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [ ] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.
### What happened?
We used the in-process etcdserver of v3client. Then we created a client, created a watch connection to the same resource every second, without freeing them, and ran it for more than 1 minute. When the client maintains a large number of watch connections, we kill the client process. After the client process is killed, when other clients attempt to establish watch connections for the same resource, the new watch connections cannot obtain new event changes.
### What did you expect to happen?
After the client is killed, the new watch connection for the same resource can properly listen to event changes.
And after analysis, the blocking problem exists. Although it is unreasonable for the client to establish a large number of watch connections with the same resource at the same time, can the etcd server do something to avoid the blocking?
### How can we reproduce it (as minimally and precisely as possible)?
We created a large number of Watch connections to the same configmap resource in a loop through a process using code similar to the following:
[main.txt](https://github.com/user-attachments/files/17710083/main.txt)
After running this program for 1 minute, kill the program. When you continue to run the kubectl get configmap -A -w command, after the configmap is modified, the configmap change cannot be watched.
### Anything else we need to know?
After the client is killed, a large number of watch connections are disconnected. The code analysis shows that the `Send()` function of `WatchCancelRequest `in `case ws := <-w.closingc` of the `(w *watchGrpcStream) run()` method in `etcd/client/v3/watch.go` is blocked and unable to continue processing.
It is suspected that a large number of `WatchCancelRequests` cause the channel in `watchGrpcStream` to be fully occupied. As a result, new `WatchResponse` cannot be pushed into `sws.ctrlStream`. The `WatchResponse` obtained from `ctrlStream` and new `WatchResponse `are blocked in `case pbresp := <-w.respc` and `case ws := <-w.closingc` in `(w *watchGrpcStream) run()`.
### Etcd version (please run commands below)
```console
$ etcd --version
# 3.5.11
$ etcdctl version
# 3.5.11
```
### Etcd configuration (command line flags or environment variables)
# paste your configuration here
### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
```console
$ etcdctl member list -w table
# paste output here
$ etcdctl --endpoints= endpoint status -w table
# paste output here
```
### Relevant log output
_No response_
Contributor guide
Research direction
Read etcd/client/v3/watch.go, focusing on watchGrpcStream.run and its closingc, respc, and ctrlStream cases. Reproduce the reported scenario with the linked main.txt program, then use kubectl get configmap -A -w while changing the resource. Done means new watch connections continue receiving event changes after many watches are disconnected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100