grpcproxy: new watchers may be added to a disconnected watchBroadcast and miss all subsequent events
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 43
Description
### Bug report criteria
- [x] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [x] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [x] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [x] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.
### What happened?
In `server/proxy/grpcproxy`, when a `watchBroadcast`'s underlying etcd watch connection is broken (the goroutine exits after `for wr := range wch` completes), the `watchBroadcast` remains in `watchBroadcasts.bcasts` with no indication that it is no longer receiving events.
This causes two problems:
1. **`add()` can place new watchers into a dead broadcast**: Since there is no `stopped` flag, `add()` happily accepts new watchers into a broadcast whose goroutine has already exited. These watchers will never receive any subsequent events.
2. **`coalesce()` can migrate watchers into a dead broadcast**: The coalesce logic only checks `nextrev` and `responses`, not whether the target broadcast is still alive. Watchers migrated to a dead broadcast will also stop receiving events.
Additionally, there is no mechanism to reassign existing watchers (orphans) from a dead broadcast to a healthy one.
### What did you expect to happen?
1. A disconnected `watchBroadcast` should be marked as stopped and refuse new watchers.
2. `coalesce()` should not migrate watchers to a stopped broadcast.
3. When a broadcast disconnects, its existing watchers should be automatically reassigned to a healthy broadcast or a newly created one.
### How can we reproduce it (as minimally and precisely as possible)?
1. Set up an etcd gRPC proxy with multiple client watchers coalesced on the same key range.
2. Cause the backend etcd watch connection to break (e.g., network partition, etcd server restart).
3. After the break, create a new watcher on the same key range through the proxy.
4. Observe that the new watcher (and any existing watchers on the dead broadcast) never receives subsequent events.
### Anything else we need to know?
_No response_
### Etcd version (please run commands below)
```console
$ etcd --version
v3.5.4
$ etcdctl version
v3.5.4
```
### Etcd configuration (command line flags or environment variables)
# paste your configuration here
### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
```console
$ etcdctl member list -w table
# paste output here
$ etcdctl --endpoints= endpoint status -w table
# paste output here
```
### Relevant log output
```Shell
```
Contributor guide
Research direction
Start in server/proxy/grpcproxy and trace watchBroadcasts.bcasts, add(), coalesce(), and the goroutine consuming wch. Reproduce a broken backend watch connection, then verify that stopped broadcasts reject new watchers, coalescing avoids them, and existing watchers are reassigned to a healthy or newly created broadcast.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100