etcd-io / etcd-io/etcd

mvcc: compacted watcher is never removed from the unsynced group when there are at least maxWatchersPerSync unsynced watchers

Open
#22,294 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
3d 1h
Merged PRs (30d)
44

Description

`watcherGroup.choose` has two branches. When the unsynced group holds fewer than
`maxWatchersPerSync` watchers it calls `chooseAll` on the group itself. When it holds at least
that many, it copies a bounded subset into a temporary group and calls `chooseAll` on the copy:

```go
func (wg *watcherGroup) choose(maxWatchers int, curRev, compactRev int64) (*watcherGroup, int64) {
if len(wg.watchers) < maxWatchers {
return wg, wg.chooseAll(curRev, compactRev)
}
ret := newWatcherGroup()
for w := range wg.watchers {
if maxWatchers <= 0 {
break
}
maxWatchers--
ret.add(w)
}
return &ret, ret.chooseAll(curRev, compactRev)
}
```

`chooseAll` is the only place that retires a watcher whose `minRev` is below `compactRev`: it
sends a `CompactRevision` response, sets `w.compacted = true`, and calls `wg.delete(w)`. In the
capped branch that `wg` is the temporary group, so the watcher is removed from the copy and left
in `s.unsynced`.

`syncWatchers` then iterates the returned group, which no longer contains the watcher, so it
never reaches `s.unsynced.delete(w)` either. Effects:

- the watcher stays in the unsynced group until the client cancels it
- every following resync selects it again and sends another `CompactRevision` response on the
same stream, once per `watchResyncPeriod` (100ms)
- `s.unsynced.size()` stays inflated, and that value is both the return of `syncWatchers` and the
input to `slowWatcherGauge`, so the resync loop's own backpressure signal is wrong

Reachable whenever at least `maxWatchersPerSync` (default 512) watchers are unsynced at the same
time and at least one of them starts below the compaction revision. That is a normal state on a
busy cluster after a compaction, and also right after a leader change or snapshot restore, since
`watchableStore.Restore` moves every synced watcher into the unsynced group at once.

Impact is availability and noise rather than data loss: no events are lost or duplicated, but a
watch that should have been cleanly retired is not, and the client receives repeated compaction
responses.

Existing coverage does not reach it. `TestWatchNoEventLossOnCompact` sets
`maxWatchersPerSync = 4` with 3 watchers, so it only ever exercises the uncapped branch. No test
in the package drives `choose` with `len(watchers) >= maxWatchers` and compacted watchers present.

Contributor guide

Open the contributing guide

Research direction

Start with watcherGroup.choose, chooseAll, and syncWatchers, then read TestWatchNoEventLossOnCompact in the watch package. Add coverage for maxWatchersPerSync or more unsynced watchers with a compacted watcher, and verify the watcher is retired once and the resync size and responses remain correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.