Unsafe recovery partially fills key range hole
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
On a 4-node TiKV cluster, we stops two nodes and then starts unsafe recovery using pd-ctl.
After unsafe recovery, we find there are lots of PD server timeout, and it turns out there is
a region fails to be created.
Failed TiKV: tikv-0 and tikv-1
Alive TiKV: tikv-2 and tikv-3
Original region ID: 1965
New region ID: 2991
Timeline:
1. 1965 on tikv-3 sends a snapshot to tikv-2.
2. Starts unsafe recovery.
3. Snapshot sent.
4. 1965 on tikv-3 becomes tombstone.
5. A peer of 1965 is created on tikv-2.
6. PD sends to tikv-2 to create 2991 to cover the key rang of 1965.
7. 2991 fails to be created because 1965 has been created on tikv-3.
8. PD considers unsafe recovery is finished.
There are actually two questions:
1. Why does PD finish unsafe recovery while there is a key rang hole?
2. Why does PD tombstone 1965 in the first place? Stoping two nodes out of
four nodes cluster should not lost replica data completely.
Note: the issue is found on a multi-rocksdb cluster. But I think it may affect single rocksdb cluster too.
Log:
* [unsafe-recovery.log](https://github.com/tikv/pd/files/12206328/unsafe-recovery.log)
* [tikv_log.tar.gz](https://github.com/tikv/pd/files/12299055/log.tar.gz)
### What did you do?
See above.
### What version of PD are you using (`pd-server -V`)?
v7.1.0
Contributor guide
Assessment
This issue has not been assessed yet.