TiFlash v7.1.6 panic when running `br restore`
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
Using br restore command to restore a chbenchmark 1500 database with TiFlash replica. There are already more than 4 TiB storage used on TiFlash instance.
2. What did you expect to see? (Required)
br restore run successfully and TiFlash replica are restored successfully too.
3. What did you see instead (Required)
During the br restore, TiFlash happen to a panic with stacktrace as follow. TiFlash panic repeatedly after this error happened.
[2025/04/23 07:09:53.408 +00:00] [FATAL] [lib.rs:497] ["[region 181708757] 181708761 ingest uuid: ? range { start: ? end: ? } length: 7104738 cf_name: \"write\" region_id: 181708757 region_epoch { conf_ver: 86 version: 2168 } cipher_iv: ?: EngineTraits(Engine(Status { code: IoError, sub_code: None, sev: NoError, state: \"IO error: No such file or directory: while stat a file for size: /data0/proxy/import/54c46afb-667d-4bd0-b8f2-68ef1c87e92f_181708757_86_2168_write.sst: No such file or directory\" }))"] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n 1: std::panicking::rust_panic_with_hook\n 2: std::panicking::begin_panic_handler::{{closure}}\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n 4: rust_begin_unwind\n 5: core::panicking::panic_fmt\n 6: raftstore::store::fsm::apply::ApplyDelegate<EK>::apply_raft_cmd\n 7: raftstore::store::fsm::apply::ApplyDelegate<EK>::process_raft_cmd\n 8: raftstore::store::fsm::apply::ApplyDelegate<EK>::handle_raft_committed_entries\n 9: <raftstore::store::fsm::apply::ApplyPoller<EK> as batch_system::batch::PollHandler<raftstore::store::fsm::apply::ApplyFsm<EK>,raftstore::store::fsm::apply::ControlFsm>>::handle_normal\n 10: batch_system::batch::Poller<N,C,Handler>::poll\n 11: std::sys_common::backtrace::__rust_begin_short_backtrace\n 12: core::ops::function::FnOnce::call_once{{vtable.shim}}\n 13: std::sys::unix::thread::Thread::new::thread_start\n 14: start_thread\n 15: thread_start\n"] [location=/workspace/source/tiflash/contrib/tiflash-proxy/components/raftstore/src/store/fsm/apply.rs:2005] [thread_name=apply-low-5]
4. What is your TiFlash version? (Required)
v7.1.6
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the panic location at tiflash-proxy/components/raftstore/src/store/fsm/apply.rs:2005 and inspect the reported missing .sst file during a br restore. Reproduce the restore against TiFlash v7.1.6 with a TiFlash replica and verify that TiFlash no longer panics repeatedly and the replica restores successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100