pd-ctl: add a region meta consistency check across PD members
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
### Background
PD followers maintain their own region meta view through the region syncer. A follower may temporarily or persistently diverge from the leader due to synchronization gaps or other failures, as described in #10666.
Follower-local read-only region HTTP APIs were enabled by #10681 and #10682. However, operators still need to query each PD member manually and compare the results themselves. `pd-ctl` does not currently provide a command that verifies whether the region meta held by all PD members is consistent.
Relying on a separate script makes deployment, authentication, versioning, and routine production use harder. This check should be available as a native `pd-ctl` command without requiring Python or another external runtime.
### Goal
Add a native command that compares region meta across the PD leader and all followers:
```bash
pd-ctl -u http://pd-0:2379 region meta-consistency
```
The command should discover all PD members from the supplied endpoint, query each member directly, and report whether their local region meta views are consistent.
### Expected behavior
The command should compare the following region meta fields:
- Region ID
- Start key and end key
- Region epoch: `conf_ver` and `version`
- Peers: peer ID, store ID, role, and witness state
- Region leader peer
The report should:
- List every inconsistent Region ID.
- Identify instances using `member-name@host:port`.
- Include all relevant Peer IDs and Store IDs.
- Report only fields that differ:
- `missing_on`
- `key_range`
- `epoch`
- `peers`
- `leader_peer`
- Avoid verbose classifications or derived fields when the raw difference is sufficient.
Example:
```json
{
"status": "inconsistent",
"summary": {
"different_regions": 1,
"by_field": {
"epoch": 1
}
},
"differences": [
{
"region_id": 42,
"epoch": {
"pd-0@10.0.0.1:2379": {
"conf_ver": 3,
"version": 8
},
"pd-1@10.0.0.2:2379": {
"conf_ver": 3,
"version": 7
}
}
}
]
}
```
Suggested exit codes:
- `0`: all PD members are consistent
- `1`: at least one stable inconsistency is confirmed
- `2`: the result is incomplete because of request failures, membership changes, unstable scans, or invalid arguments
### Suggested implementation
Register `meta-consistency` under the existing `region` command.
1. Query `/pd/api/v1/members` and determine each member's name, client URL, and current role.
2. Send requests directly to each member endpoint. Do not use a client path that may automatically switch to another PD endpoint.
3. Set `PD-Allow-Follower-Handle: true` so followers serve Region data from their local synchronized cache.
4. Scan through `/pd/api/v1/regions/key` using a small configurable batch size and the previous batch's end key as the next cursor.
5. Query `/pd/api/v1/regions/count` before and after each scan. Retry the whole cluster scan if any member's count changes.
6. Normalize only the region meta fields being compared. Ignore heartbeat statistics such as read/write bytes, pending peers, and approximate size.
7. Keep scan state bounded. Use a temporary disk-backed store instead of retaining every member's complete region meta in memory.
8. Compare records by Region ID and stream the final JSON report instead of building all differences in memory.
9. Recheck a bounded number of differing Region IDs through `/pd/api/v1/region/id/{id}` after a short delay to distinguish stable divergence from transient scan skew.
10. Verify that membership and the PD leader did not change during the scan.
The targeted confirmation limit must only limit additional requests. It must not truncate the final list of inconsistent Regions.
### Production safety
Use conservative defaults:
```text
batch size: 128 Regions
request interval: 50 ms
request timeout: 10 s
global concurrency: 1
confirmation limit: 128 Regions
```
The command should:
- Use small, repeated HTTP scans instead of `/regions`.
- Avoid concurrent scans against multiple PD members.
- Avoid issuing Region lookup gRPC requests.
- Reuse persistent HTTP connections.
- Inherit the existing `pd-ctl` TLS options.
- Write progress messages to stderr and keep stdout as valid JSON.
### Tests
Add Go tests using three local HTTP test servers to cover:
- Fully consistent region meta.
- Missing Regions on one member.
- Key range divergence.
- Epoch divergence.
- Peer divergence.
- Region leader peer divergence.
- Three members with different values and no majority.
- Transient differences resolved during confirmation.
- Bounded confirmation requests with a large difference set.
- Large unsigned Region IDs and epochs.
- Follower-local headers, pagination, and request limits.
Also run an end-to-end test with a three-PD TiUP cluster, inject Region heartbeats, and verify that the native `pd-ctl` command reports a consistent cluster.
### Non-goals
- Repairing or deleting stale region meta.
- Modifying PD Server APIs.
- Providing an atomic cluster-wide snapshot.
- Increasing scan concurrency for faster completion.
Contributor guide
Research direction
Start at the existing region command in pd-ctl and review the listed PD member, region, and region-count HTTP endpoints. Add Go tests with three local HTTP test servers covering consistency, divergence, confirmation, pagination, and request limits. Done means the command emits valid JSON, preserves the specified exit codes and safety limits, and passes the three-PD TiUP end-to-end test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100