LoopWatcher can miss changes after etcd compaction
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
### What did you do?
Disconnect a LoopWatcher, update its watched key, advance the etcd
revision, compact past the missed update, and then reconnect the watcher.
This was found while investigating:
https://github.com/tidbcloud/pd-cse/issues/559
### What did you expect to see?
After detecting a compacted watch revision, LoopWatcher should reload a
consistent snapshot and continue watching from the snapshot revision plus
one.
### What did you see instead?
LoopWatcher changes its watch revision to the compact revision without
reloading. Changes in the compacted interval are not replayed, so the
consumer cache can remain stale indefinitely.
Paginated loading can also read different revisions, and an earlier
callback error can be overwritten by a later successful callback.
### What version of PD are you using (`pd-server -V`)?
master at 39b622049
Contributor guide
Research direction
Start by locating the LoopWatcher entry point and its reconnect, revision, paginated-loading, and callback paths. Reproduce the disconnect, update, revision advance, compaction, and reconnect sequence described in the report. Done means a compacted watch reloads one consistent snapshot, resumes from snapshot revision plus one, and preserves earlier callback errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100