TiFlash crashes after "Start cancel pre-handling from upper layer"

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

TiFlash accidentally encountered a crash. After restart, the crash didn't happen again.

[2025/04/22 05:56:54.312 +00:00] [INFO] [ApplySnapshot.cpp:299] ["Begin apply snapshot, new_region=[region_id=268137913 applied_term=7 applied_index=15]"] [thread_id=139]
[2025/04/22 05:56:54.312 +00:00] [INFO] [ApplySnapshot.cpp:75] ["[region_id=268137913 applied_term=5 applied_index=5] set state to `Applying`"] [thread_id=139]
[2025/04/22 05:56:54.312 +00:00] [WARN] [KVStore.cpp:191] ["tryFlushRegionCacheInStorage can not get table, region [region_id=268137913 applied_term=5 applied_index=5] table_id=1683, ignored"] [thread_id=139]
[2025/04/22 05:56:54.312 +00:00] [INFO] [KVStore.cpp:373] ["Start to persist [region_id=268137913 applied_term=5 applied_index=5], cache size: 0 bytes for `save previous region before apply `"] [thread_id=139]
[2025/04/22 05:56:54.337 +00:00] [INFO] [RegionTable.cpp:412] ["internal region has larger range, keyspace=4294967295 table_id=1683 region_id=268137913"] [thread_id=139]
[2025/04/22 05:56:54.356 +00:00] [INFO] [KVStore.cpp:373] ["Start to persist [region_id=268137913 applied_term=7 applied_index=15], cache size: 0 bytes for `save current region after apply `"] [thread_id=139]
[2025/04/22 05:56:54.361 +00:00] [INFO] [ApplySnapshot.cpp:312] ["Finish apply snapshot, cost=0.049s region_id=268137913"] [thread_id=139]
[2025/04/22 05:56:54.361 +00:00] [INFO] [PrehandleSnapshot.cpp:747] ["Start cancel pre-handling from upper layer, region_id=268129202"] [thread_id=139]
[2025/04/22 05:56:54.361 +00:00] [INFO] [PrehandleSnapshot.cpp:771] ["Release prehandled snapshot, clean 0 dmfiles, region_id=268129202 keyspace=4294967295 table_id=1683"] [thread_id=139]Show context
[2025/04/22 05:56:54.362 +00:00] [ERROR] [BaseDaemon.cpp:370] [########################################] [source=BaseDaemon] [thread_id=567]
[2025/04/22 05:56:54.362 +00:00] [ERROR] [BaseDaemon.cpp:371] ["(from thread 139) Received signal Segmentation fault(11)."] [source=BaseDaemon] [thread_id=567]
[2025/04/22 05:56:54.362 +00:00] [ERROR] [BaseDaemon.cpp:401] ["Address: 0x110"] [source=BaseDaemon] [thread_id=567]
[2025/04/22 05:56:54.362 +00:00] [ERROR] [BaseDaemon.cpp:416] ["Address not mapped to object."] [source=BaseDaemon] [thread_id=567]
[2025/04/22 05:56:54.435 +00:00] [ERROR] [BaseDaemon.cpp:563] ["\n       0x6acd1bc\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+111989180]\n                \tlibs/libdaemon/src/BaseDaemon.cpp:214\n  0xffff909e8820\t<unknown symbol> [linux-vdso.so.1+2080]\n       0x72cd968\tDB::StorageDeltaMerge::getAndMaybeInitStore(DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >*) [tiflash+120379752]\n                \tdbms/src/Storages/StorageDeltaMerge.cpp:1936\n       0x7c07110\tvoid DB::KVStore::releasePreHandledSnapshot<DB::RegionPtrWithSnapshotFiles>(DB::RegionPtrWithSnapshotFiles const&, DB::TMTContext&) [tiflash+130052368]\n                \tdbms/src/Storages/KVStore/MultiRaft/PrehandleSnapshot.cpp:773\n       0x7be5158\tReleasePreHandledSnapshot [tiflash+129913176]\n                \tdbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:738\n  0xffff8e136780\t_$LT$engine_store_ffi..observer..TiFlashObserver$LT$T$C$ER$GT$$u20$as$u20$raftstore..coprocessor..ApplySnapshotObserver$GT$::cancel_apply_snapshot::h397b71c26d677a8f [libtiflash_proxy.so+25823104]\n  0xffff8efa2e6c\traftstore::store::worker::region::Runner$LT$EK$C$R$C$T$GT$::handle_pending_applies::h05bf95f1570f6c67 [libtiflash_proxy.so+40947308]\n  0xffff8e653c30\tyatp::task::future::RawTask$LT$F$GT$::poll::h15383bccb4d932ed [libtiflash_proxy.so+31185968]\n  0xffff8fd3a214\t_$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::h5adbfadd82cb614e [libtiflash_proxy.so+55198228]\n  0xffff8e22e724\tstd::sys_common::backtrace::__rust_begin_short_backtrace::hcea1723f02df148f [libtiflash_proxy.so+26838820]\n  0xffff8e265bc4\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h3aa4e6cbbdc368f4 [libtiflash_proxy.so+27065284]\n  0xffff8f5a13c0\tstd::sys::unix::thread::Thread::new::thread_start::hcc916efc5918f503 [libtiflash_proxy.so+47231936]\n  0xffff8c5556b8\tstart_thread [libc.so.6+526008]"] [source=BaseDaemon] [thread_id=567]

[2025/04/22 05:57:20.202 +00:00] [INFO] [SchemaBuilder.cpp:1208] ["Create table db_124.bsc_token_latest_balance end, database_id=124 table_id=1683 action=SyncAllSchema"] [source="keyspace=4294967295"] [thread_id=20]Show context
[2025/04/22 05:57:20.198 +00:00] [INFO] [SchemaBuilder.cpp:1174] ["Create table db_124.bsc_token_latest_balance (database_id=124 table_id=1683) with statement: CREATE TABLE `db_124`.`t_1683`(`chainid` String, `address` String, `contractaddress` String, `height` Nullable(Int64), `blockts` Nullable(Int64), `balance` Nullable(String), `msgts` Nullable(Int64), `name` Nullable(String), `symbol` Nullable(String), `decimals` Nullable(Int32), `dt` Nullable(MyDate), `db_create_time` MyDateTime(0), `db_modify_time` MyDateTime(0), `_tidb_rowid` String) Engine = DeltaMerge((`_tidb_rowid`), 'xxx', 0)"] [source="keyspace=4294967295"] [thread_id=20]
[2025/04/22 05:57:20.198 +00:00] [INFO] [SchemaBuilder.cpp:822] ["Storage instance does not exist, tryRecoverPhysicalTable is ignored, table_id=1683 action=SyncAllSchema"] [source="keyspace=4294967295"] [thread_id=20]
[2025/04/22 05:57:20.198 +00:00] [INFO] [SchemaBuilder.cpp:1140] ["Create table db_124.bsc_token_latest_balance begin, database_id=124, table_id=1683 action=SyncAllSchema"] [source="keyspace=4294967295"] [thread_id=20]
[2025/04/22 05:57:20.198 +00:00] [INFO] [SchemaBuilder.cpp:1407] ["Table w3w_dw.bsc_token_latest_balance syncing during sync all schemas, database_id=124 table_id=1683"] [source="keyspace=4294967295"] [thread_id=20]

4. What is your TiFlash version? (Required)

v7.5.6

Start with the crash path in dbms/src/Storages/StorageDeltaMerge.cpp:1936, then trace releasePreHandledSnapshot in dbms/src/Storages/KVStore/MultiRaft/PrehandleSnapshot.cpp:773 and ReleasePreHandledSnapshot in dbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:738. Compare the cancellation log sequence with the v7.5.6 stack trace and determine why the snapshot cancellation reaches the segmentation fault. Done means the cancellation path no longer crashes under the reported sequence.