pingcap / pingcap/tiflash

TiFlash panic if an Region across multiple table_id or index keyrange

Open
#10,908 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/storage type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)
  1. Generate a Region that across multiple table_id or index keyrange. With
{"id":1560448274,"start_key":"7480000000002874FF3E00000000000000F8","end_key":"7480000000002874FFA55F698000000000FF0000010400000000FF0009EEB500000000FB"}
  1. Manually add a Region learner peer to tiflash using pd-ctl
pd-ctl -u 127.0.0.1:2379 operator add add-learner 1560448274 36
2. What did you expect to see? (Required)
3. What did you see instead (Required)

tiflash panic

[2026/06/22 13:12:06.248 +00:00] [FATAL] [Exception.cpp:96] ["Code: 49, e.displayText() = DB::Exception: Illegal region range, should not happen, start_key=? end_key=?, e.what() = DB::Exception, Stack trace:
  0xaaaae5344cc4    DB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&) [tiflash+79252676]
  0xaaaae8ff9e78    DB::RegionRangeKeys::RegionRangeKeys(DB::StringObject<true>&&, DB::StringObject<true>&&) [tiflash+142909048]
                    dbms/src/Storages/KVStore/MultiRaft/RegionState.cpp:149
  0xaaaae8ff9310    DB::RegionState::updateRegionRange() [tiflash+142906128]
                    /usr/local/bin/../include/c++/v1/__memory/construct_at.h:41
  0xaaaae8fe8878    DB::RegionMeta::RegionMeta(metapb::Peer, metapb::Region, raft_serverpb::RaftApplyState) [tiflash+142837880]
                    dbms/src/Storages/KVStore/MultiRaft/RegionMeta.cpp:495
  0xaaaae8f24e98    DB::KVStore::genRegionPtr(metapb::Region&&, unsigned long, unsigned long, unsigned long) [tiflash+142036632]
                    dbms/src/Storages/KVStore/KVStore.cpp:495
  0xaaaae8f76338    PreHandleSnapshot [tiflash+142369592]
                    dbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:685
  0xffff8cf5eb64    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::hf1ec63b2511a1357 [libtiflash_proxy.so+63744868]
                    contrib/tiflash-proxy/proxy_components/proxy_ffi/src/engine_store_helper_impls.rs:292
  0xffff8ca7c978    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf91bcd6c2b69826b [libtiflash_proxy.so+58624376]
                    contrib/tiflash-proxy/proxy_components/engine_store_ffi/src/core/forward_raft/snapshot.rs:97
  0xffff8d81eaa4    yatp::task::future::RawTask$LT$F$GT$::poll::h2879d332e78fab98 [libtiflash_proxy.so+72919716]
                    /workspace/.cargo/git/checkouts/yatp-e704b73c3ee279b6/793be4d/src/task/future.rs:59
  0xffff8f21b6d8    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::h5569b38cc95e4640 [libtiflash_proxy.so+100169432]
                    /workspace/.cargo/git/checkouts/yatp-e704b73c3ee279b6/793be4d/src/task/future.rs:387
  0xffff8f20b010    std::sys_common::backtrace::__rust_begin_short_backtrace::h9b780a3f40586e98 [libtiflash_proxy.so+100102160]
                    /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:155
  0xffff8f20cec4    core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hd3739b609fe7651a [libtiflash_proxy.so+100110020]
                    /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250
  0xffff8e864d18    std::sys::unix::thread::Thread::new::thread_start::h35e031273648d290 [libtiflash_proxy.so+89984280]
                    /root/.rustup/toolchains/nightly-2023-12-28-aarch64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:108
  0xffff88f4ab40    start_thread [libc.so.6+527168]
  0xffff88fb515c    thread_start [libc.so.6+962908]"] [source="RawCppPtr DB::PreHandleSnapshot(EngineStoreServerWrap *, BaseBuffView, uint64_t, SSTViewVec, uint64_t, uint64_t)"] [thread_id=4581]
4. What is your TiFlash version? (Required)

v8.5.4

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure by creating the Region described and adding a learner with pd-ctl. Start at dbms/src/Storages/KVStore/MultiRaft/RegionState.cpp:149 and follow the snapshot path through RegionMeta.cpp:495, KVStore.cpp:495, ProxyFFI.cpp:685, and the Rust snapshot entry point. Done means a Region spanning multiple table_id or index keyranges no longer causes TiFlash to panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, rust
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.