tikv / tikv/pd

LoopWatcher can miss changes after etcd compaction

Open
#11,032 1 comment 0 reactions 0 assignees View on GitHub
severity/moderate type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?

Disconnect a LoopWatcher, update its watched key, advance the etcd
revision, compact past the missed update, and then reconnect the watcher.

This was found while investigating:
https://github.com/tidbcloud/pd-cse/issues/559

### What did you expect to see?

After detecting a compacted watch revision, LoopWatcher should reload a
consistent snapshot and continue watching from the snapshot revision plus
one.

### What did you see instead?

LoopWatcher changes its watch revision to the compact revision without
reloading. Changes in the compacted interval are not replayed, so the
consumer cache can remain stale indefinitely.

Paginated loading can also read different revisions, and an earlier
callback error can be overwritten by a later successful callback.

### What version of PD are you using (`pd-server -V`)?

master at 39b622049

Contributor guide

Open the contributing guide

Research direction

Start by locating the LoopWatcher entry point and its reconnect, revision, paginated-loading, and callback paths. Reproduce the disconnect, update, revision advance, compaction, and reconnect sequence described in the report. Done means a compacted watch reloads one consistent snapshot, resumes from snapshot revision plus one, and preserves earlier callback errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.