pingcap / pingcap/tiflash

TiFlash v7.1.6 panic when running `br restore`

Open
#10,141 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

affects-6.5 affects-7.1 affects-7.5 component/storage impact/panic severity/major type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Using br restore command to restore a chbenchmark 1500 database with TiFlash replica. There are already more than 4 TiB storage used on TiFlash instance.

2. What did you expect to see? (Required)

br restore run successfully and TiFlash replica are restored successfully too.

3. What did you see instead (Required)

During the br restore, TiFlash happen to a panic with stacktrace as follow. TiFlash panic repeatedly after this error happened.

[2025/04/23 07:09:53.408 +00:00] [FATAL] [lib.rs:497] ["[region 181708757] 181708761 ingest uuid: ? range { start: ? end: ? } length: 7104738 cf_name: \"write\" region_id: 181708757 region_epoch { conf_ver: 86 version: 2168 } cipher_iv: ?: EngineTraits(Engine(Status { code: IoError, sub_code: None, sev: NoError, state: \"IO error: No such file or directory: while stat a file for size: /data0/proxy/import/54c46afb-667d-4bd0-b8f2-68ef1c87e92f_181708757_86_2168_write.sst: No such file or directory\" }))"] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}\n   1: std::panicking::rust_panic_with_hook\n   2: std::panicking::begin_panic_handler::{{closure}}\n   3: std::sys_common::backtrace::__rust_end_short_backtrace\n   4: rust_begin_unwind\n   5: core::panicking::panic_fmt\n   6: raftstore::store::fsm::apply::ApplyDelegate<EK>::apply_raft_cmd\n   7: raftstore::store::fsm::apply::ApplyDelegate<EK>::process_raft_cmd\n   8: raftstore::store::fsm::apply::ApplyDelegate<EK>::handle_raft_committed_entries\n   9: <raftstore::store::fsm::apply::ApplyPoller<EK> as batch_system::batch::PollHandler<raftstore::store::fsm::apply::ApplyFsm<EK>,raftstore::store::fsm::apply::ControlFsm>>::handle_normal\n  10: batch_system::batch::Poller<N,C,Handler>::poll\n  11: std::sys_common::backtrace::__rust_begin_short_backtrace\n  12: core::ops::function::FnOnce::call_once{{vtable.shim}}\n  13: std::sys::unix::thread::Thread::new::thread_start\n  14: start_thread\n  15: thread_start\n"] [location=/workspace/source/tiflash/contrib/tiflash-proxy/components/raftstore/src/store/fsm/apply.rs:2005] [thread_name=apply-low-5]
4. What is your TiFlash version? (Required)

v7.1.6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the panic location at tiflash-proxy/components/raftstore/src/store/fsm/apply.rs:2005 and inspect the reported missing .sst file during a br restore. Reproduce the restore against TiFlash v7.1.6 with a TiFlash replica and verify that TiFlash no longer panics repeatedly and the replica restores successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.