[tidb-x] After killing a tikv for some time, a tiflash wn repeatedly crashes

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run tpch
2、kill one of tikv
3、after some time，a tiflash wn repeatedly crashes

2. What did you expect to see? (Required)

tiflash run normally

3. What did you see instead (Required)

After killing a tikv for some time, a tiflash wn repeatedly crashes
{"log":"2026-01-20T14:08:12.44876661+08:00 stderr F [2026/01/20 14:08:12.448 +08:00] [FATAL] [lib.rs:556] ["called Result::unwrap() on an Err value: Other(\"[components/kvengine/src/shard.rs:438]: errors is not empty: [DfsError(NoSuchKey(\\\"file 463686552158470393.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470394.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470389.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686667396710805.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470387.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686705433804809.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686627747430465.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552000921655.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686590510137720.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470390.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470388.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470395.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470391.sst not exist,\\\")), DfsError(NoSuchKey(\\\"file 463686552158470392.sst not exist,\\\"))]\")"] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/alloc/src/boxed.rs:2029:9\n std::panicking::rust_panic_with_hook\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:783:13\n 2: std::panicking::begin_panic_handler::{{closure}}\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:657:13\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/backtrace.rs:171:18\n 4: rust_begin_unwind\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/panicking.rs:645:5\n 5: core::panicking::panic_fmt\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/panicking.rs:72:14\n 6: core::result::unwrap_failed\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/result.rs:1649:5\n 7: proxy_ffi::cloud_helper::CloudHelper::get_shard\n 8: proxy_ffi::cloud_helper::CloudHelper::make_sst_reader\n 9: proxy_ffi::sst_reader_impls::ffi_make_sst_reader\n 10: _ZN2DB13MonoSSTReaderC1EPKNS_22TiFlashRaftProxyHelperENS_7SSTViewENSt3__110shared_ptrIKNS_15RegionRangeKeysEEERKNS6_INS_6LoggerEEE\n at workspace/source/tiflash/dbms/src/Storages/KVStore/FFI/SSTReader.cpp:114:13\n 11: _ZNSt3__110__function6__funcIZN2DB2DM26SSTFilesToBlockInputStreamC1ENS_10shared_ptrINS2_6RegionEEEmRKNS2_10SSTViewVecEPKNS2_22TiFlashRaftProxyHelperERNS2_10TMTContextEONS_8optionalINS3_16SSTScanSoftLimitEEENS5_INS2_16PreHandlingTrace4ItemEEEONS3_30SSTFilesToBlockInputStreamOptsEE3$0NS_9allocatorISP_EEFNS_10unique_ptrINS2_13MonoSSTReaderENS_14default_deleteIST_EEEESD_NS2_7SSTViewENS5_IKNS2_15RegionRangeKeysEEERKNS5_INS2_6LoggerEEEEEclEOSD_OSX_OS10_S14.387a1388c4b5c8fe1da21e49667395e6\n at usr/local/bin/../include/c++/v1/__memory/unique_ptr.h:689:30\n 12: _ZNSt3__111make_uniqueB8ue170006IN2DB14MultiSSTReaderINS1_13MonoSSTReaderENS1_7SSTViewEEEJRPKNS1_22TiFlashRaftProxyHelperENS1_16ColumnFamilyTypeERZNS1_2DM26SSTFilesToBlockInputStreamC1ENS_10shared_ptrINS1_6RegionEEEmRKNS1_10SSTViewVecES8_RNS1_10TMTContextEONS_8optionalINSB_16SSTScanSoftLimitEEENSD_INS1_16PreHandlingTrace4ItemEEEONSB_30SSTFilesToBlockInputStreamOptsEE3$0RNS_6vectorIS4_NS_9allocatorIS4_EEEERNSD_INS1_6LoggerEEENSD_IKNS1_15RegionRangeKeysEEEEEENS_11__unique_ifIT_E15__unique_singleEDpOT0\n at usr/local/bin/../include/c++/v1/__functional/function.h:517:16\n 13: _ZN2DB2DM26SSTFilesToBlockInputStreamC1ENSt3__110shared_ptrINS_6RegionEEEmRKNS_10SSTViewVecEPKNS_22TiFlashRaftProxyHelperERNS_10TMTContextEONS2_8optionalINS0_16SSTScanSoftLimitEEENS3_INS_16PreHandlingTrace4ItemEEEONS0_30SSTFilesToBlockInputStreamOptsE\n at workspace/source/tiflash/dbms/src/Storages/DeltaMerge/Decode/SSTFilesToBlockInputStream.cpp💯27\n 14: _ZN2DB7KVStore22preHandleSSTsToDTFilesENSt3__110shared_ptrINS_6RegionEEENS_10SSTViewVecEmmNS_2DM18FileConvertJobTypeERNS_10TMTContextE\n at usr/local/bin/../include/c++/v1/__memory/construct_at.h:41:46\n 15: _ZN2DB7KVStore24preHandleSnapshotToFilesENSt3__110shared_ptrINS_6RegionEEENS_10SSTViewVecEmmNS1_8optionalImEERNS_10TMTContextE\n at workspace/source/tiflash/dbms/src/Storages/KVStore/MultiRaft/PrehandleSnapshot.cpp:292:34\n 16: PreHandleSnapshot\n at workspace/source/tiflash/dbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:697:42\n 17: proxy_ffi::engine_store_helper_impls::::pre_handle_snapshot\n 18: engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl\n 19: yatp::task::future::RawTask::poll\n 20: <yatp::task::future::Runner as yatp::pool:🏃:Runner>::handle\n 21: std::sys_common::backtrace::__rust_begin_short_backtrace\n 22: core::ops::function::FnOnce::call_once{{vtable.shim}}\n 23: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/alloc/src/boxed.rs:2015:9\n <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/alloc/src/boxed.rs:2015:9\n std::sys::unix:🧵:Thread:🆕:thread_start\n at rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys/unix/thread.rs:108:17\n 24: start_thread\n 25: __GI___clone\n"] [location=proxy_components/proxy_ffi/src/cloud_helper.rs:269] [thread_name=region-task-3]","container":"tiflash","namespace":"ha-test-tiflash-tps-8047826-1-764","pod":"tc-tiflash-0"}

4. What is your TiFlash version? (Required)

TiFlash
Release Version: v8.5.4-nextgen.202510.8
Edition: Enterprise
Git Commit Hash: b23fae4890177ef62b8656f0cceade182287dfd0
Git Branch: HEAD
UTC Build Time: 2026-01-13 01:04:19
Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto next-gen hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake
Profile: RELWITHDEBINFO
Compiler: clang++ 17.0.6
Raft Proxy
Git Commit Hash: 2505f2f8d3061d8e61aa6f4ff4b91ade95a50785
Git Commit Branch: HEAD
UTC Build Time: ""
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: "raftstore-proxy/external-jemalloc" portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
2026-01-19T16:03:30.184+0800

Reproduce the TPCH scenario by killing one TiKV and observe the repeated TiFlash crash. Start with proxy_components/proxy_ffi/src/cloud_helper.rs at CloudHelper::get_shard and the errors reported from components/kvengine/src/shard.rs:438, then trace the callers in SSTReader.cpp, SSTFilesToBlockInputStream.cpp, and PrehandleSnapshot.cpp. Done means TiFlash continues running after the TiKV failure without the repeated missing-SST crash.