tikv / tikv/pd

Unsafe recovery partially fills key range hole

Open
#6,859 3 comments 0 reactions 0 assignees View on GitHub
affects-7.5 affects-8.1 affects-8.5 severity/major type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

On a 4-node TiKV cluster, we stops two nodes and then starts unsafe recovery using pd-ctl.
After unsafe recovery, we find there are lots of PD server timeout, and it turns out there is
a region fails to be created.

Failed TiKV: tikv-0 and tikv-1
Alive TiKV: tikv-2 and tikv-3
Original region ID: 1965
New region ID: 2991

Timeline:

1. 1965 on tikv-3 sends a snapshot to tikv-2.
2. Starts unsafe recovery.
3. Snapshot sent.
4. 1965 on tikv-3 becomes tombstone.
5. A peer of 1965 is created on tikv-2.
6. PD sends to tikv-2 to create 2991 to cover the key rang of 1965.
7. 2991 fails to be created because 1965 has been created on tikv-3.
8. PD considers unsafe recovery is finished.

There are actually two questions:

1. Why does PD finish unsafe recovery while there is a key rang hole?
2. Why does PD tombstone 1965 in the first place? Stoping two nodes out of
four nodes cluster should not lost replica data completely.

Note: the issue is found on a multi-rocksdb cluster. But I think it may affect single rocksdb cluster too.

Log:
* [unsafe-recovery.log](https://github.com/tikv/pd/files/12206328/unsafe-recovery.log)
* [tikv_log.tar.gz](https://github.com/tikv/pd/files/12299055/log.tar.gz)

### What did you do?

See above.

### What version of PD are you using (`pd-server -V`)?

v7.1.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.