pingcap / pingcap/tidb-operator
tikv crash during volumerestore [restore for 5577]
Open
@BornChanger is already working on this.
Since Oct 25, 2024.
area/ebs-br
type/bug
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 540
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 18
Description
Bug Report
What version of Kubernetes are you using?
What version of TiDB Operator are you using?
What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?
What's the status of the TiDB cluster pods?
What did you do?
- do volumebackup
- do volumerestore
What did you expect to see?
- volumerestore success
What did you see instead?
- volumerestore failed
- Confirmed corruption is likely in rocksdb layer, rather than EBS snapshots. Corrupt log file was generated (and last fsynced as per manifest), 15 seconds before ebs snapshots started.
There is some unknown write path which does partial writes to wal file, without updating the manifest.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.