TiFlash panic if an Region across multiple table_id or index keyrange
Open
Nobody has claimed this yet.
component/storage
type/bug
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
- Generate a Region that across multiple table_id or index keyrange. With
{"id":1560448274,"start_key":"7480000000002874FF3E00000000000000F8","end_key":"7480000000002874FFA55F698000000000FF0000010400000000FF0009EEB500000000FB"}
- Manually add a Region learner peer to tiflash using pd-ctl
pd-ctl -u 127.0.0.1:2379 operator add add-learner 1560448274 36
2. What did you expect to see? (Required)
3. What did you see instead (Required)
tiflash panic
[2026/06/22 13:12:06.248 +00:00] [FATAL] [Exception.cpp:96] ["Code: 49, e.displayText() = DB::Exception: Illegal region range, should not happen, start_key=? end_key=?, e.what() = DB::Exception, Stack trace:
0xaaaae5344cc4 DB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&) [tiflash+79252676]
0xaaaae8ff9e78 DB::RegionRangeKeys::RegionRangeKeys(DB::StringObject<true>&&, DB::StringObject<true>&&) [tiflash+142909048]
dbms/src/Storages/KVStore/MultiRaft/RegionState.cpp:149
0xaaaae8ff9310 DB::RegionState::updateRegionRange() [tiflash+142906128]
/usr/local/bin/../include/c++/v1/__memory/construct_at.h:41
0xaaaae8fe8878 DB::RegionMeta::RegionMeta(metapb::Peer, metapb::Region, raft_serverpb::RaftApplyState) [tiflash+142837880]
dbms/src/Storages/KVStore/MultiRaft/RegionMeta.cpp:495
0xaaaae8f24e98 DB::KVStore::genRegionPtr(metapb::Region&&, unsigned long, unsigned long, unsigned long) [tiflash+142036632]
dbms/src/Storages/KVStore/KVStore.cpp:495
0xaaaae8f76338 PreHandleSnapshot [tiflash+142369592]
dbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:685
0xffff8cf5eb64 proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::hf1ec63b2511a1357 [libtiflash_proxy.so+63744868]
contrib/tiflash-proxy/proxy_components/proxy_ffi/src/engine_store_helper_impls.rs:292
0xffff8ca7c978 engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf91bcd6c2b69826b [libtiflash_proxy.so+58624376]
contrib/tiflash-proxy/proxy_components/engine_store_ffi/src/core/forward_raft/snapshot.rs:97
0xffff8d81eaa4 yatp::task::future::RawTask$LT$F$GT$::poll::h2879d332e78fab98 [libtiflash_proxy.so+72919716]
/workspace/.cargo/git/checkouts/yatp-e704b73c3ee279b6/793be4d/src/task/future.rs:59
0xffff8f21b6d8 _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::h5569b38cc95e4640 [libtiflash_proxy.so+100169432]
/workspace/.cargo/git/checkouts/yatp-e704b73c3ee279b6/793be4d/src/task/future.rs:387
0xffff8f20b010 std::sys_common::backtrace::__rust_begin_short_backtrace::h9b780a3f40586e98 [libtiflash_proxy.so+100102160]
/root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:155
0xffff8f20cec4 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hd3739b609fe7651a [libtiflash_proxy.so+100110020]
/root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250
0xffff8e864d18 std::sys::unix::thread::Thread::new::thread_start::h35e031273648d290 [libtiflash_proxy.so+89984280]
/root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:108
0xffff88f4ab40 start_thread [libc.so.6+527168]
0xffff88fb515c thread_start [libc.so.6+962908]"] [source="RawCppPtr DB::PreHandleSnapshot(EngineStoreServerWrap *, BaseBuffView, uint64_t, SSTViewVec, uint64_t, uint64_t)"] [thread_id=4581]
4. What is your TiFlash version? (Required)
v8.5.4
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure by creating the Region described and adding a learner with pd-ctl. Start at dbms/src/Storages/KVStore/MultiRaft/RegionState.cpp:149 and follow the snapshot path through RegionMeta.cpp:495, KVStore.cpp:495, ProxyFFI.cpp:685, and the Rust snapshot entry point. Done means a Region spanning multiple table_id or index keyranges no longer causes TiFlash to panic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, rust
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100