TiFlash down occasionally when profiling in rhel 8

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

[2022/07/01 16:01:47.430 +08:00] [ERROR] [WriteBatches.h:69] ["WriteBatches:!!!=========================Modifications in log haven't persisted=========================!!! Stack trace: \n       0x1d272d3\tStackTrace::StackTrace() [tiflash+30569171]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0x77aa7bf\tDB::DM::WriteBatches::~WriteBatches()::'lambda'(DB::WriteBatch const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::operator()(DB::WriteBatch const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+125478847]\n                \tdbms/src/Storages/DeltaMerge/WriteBatches.h:65\n       0x77a6b47\tDB::DM::WriteBatches::~WriteBatches() [tiflash+125463367]\n                \tdbms/src/Storages/DeltaMerge/WriteBatches.h:72\n       0x78abafd\tDB::DM::DeltaValueSpace::flush(DB::DM::DMContext&) [tiflash+126532349]\n                \tdbms/src/Storages/DeltaMerge/Delta/DeltaValueSpace.cpp:207\n       0x77c405a\tDB::DM::Segment::flushCache(DB::DM::DMContext&) [tiflash+125583450]\n                \tdbms/src/Storages/DeltaMerge/Segment.cpp:1328\n       0x77900a4\tDB::DM::DeltaMergeStore::flushCache(std::__1::shared_ptr<DB::DM::DMContext> const&, DB::DM::RowKeyRange const&) [tiflash+125370532]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:1008\n       0x1e6aa4f\tDB::DM::DeltaMergeStore::flushCache(DB::Context const&, DB::DM::RowKeyRange const&) [tiflash+31894095]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.h:374\n       0x78ffd7b\tDB::KVStore::tryFlushRegionCacheInStorage(DB::TMTContext&, DB::Region const&, Poco::Logger*) [tiflash+126877051]\n                \tdbms/src/Storages/Transaction/KVStore.cpp:153\n       0x790284a\tDB::KVStore::handleUselessAdminRaftCmd(raft_cmdpb::AdminCmdType, unsigned long, unsigned long, unsigned long, DB::TMTContext&) [tiflash+126888010]\n                \tdbms/src/Storages/Transaction/KVStore.cpp:383\n       0x7902a2a\tDB::KVStore::handleAdminRaftCmd(raft_cmdpb::AdminRequest&&, raft_cmdpb::AdminResponse&&, unsigned long, unsigned long, unsigned long, DB::TMTContext&) [tiflash+126888490]\n                \tdbms/src/Storages/Transaction/KVStore.cpp:410\n       0x791b925\tHandleAdminRaftCmd [tiflash+126990629]\n                \tdbms/src/Storages/Transaction/ProxyFFI.cpp:114\n  0x7fa0a816fbc0\traftstore::engine_store_ffi::_$LT$impl$u20$raftstore..engine_store_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::handle_admin_raft_cmd::hc999c7ac97c80db7 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+20614080]\n  0x7fa0a8910841\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::exec_admin_cmd::hc9d6a09e7d1209ef [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28612673]\n  0x7fa0a89042e9\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::process_raft_cmd::h4c8bba241d5ee4d4 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28562153]\n  0x7fa0a8902d6c\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::handle_raft_committed_entries::h478c4bad834ddec7 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28556652]\n  0x7fa0a8922f91\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_apply::he4c0b1137ae7cad9 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28688273]\n  0x7fa0a8925bfe\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_tasks::h5b138e5889c69e66 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28699646]\n  0x7fa0a8928510\t_$LT$raftstore..store..fsm..apply..ApplyPoller$LT$EK$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..apply..ApplyFsm$LT$EK$GT$$C$raftstore..store..fsm..apply..ControlFsm$GT$$GT$::handle_normal::he20c24833a37424f [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28710160]\n  0x7fa0a859b609\tbatch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::hec57674a22fdcc30 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+24987145]\n  0x7fa0a8c44cd8\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h189c0b2868fa537c [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+31972568]\n  0x7fa0a847cec1\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h7c53021c905641b6 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+23813825]\n  0x7fa0a8e5afca\tstd::sys::unix::thread::Thread::new::thread_start::hd39c5f08bdcda277 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+34160586]\n  0x7fa0a64a4ea5\tstart_thread [libpthread.so.0+32421]\n  0x7fa0a5fb796d\t__clone [libc.so.6+1042797]"] [thread_id=43]
[2022/07/01 16:01:47.450 +08:00] [ERROR] [Exception.cpp:85] ["DB::EngineStoreApplyRes DB::HandleAdminRaftCmd(const DB::EngineStoreServerWrap *, DB::BaseBuffView, DB::BaseBuffView, DB::RaftCmdHeader):Code: 75, e.displayText() = DB::Exception: Cannot write to file /data0/db/data/t_79/log/page_209_0/page,[truncate_res = -1],[errno_after_truncate = Invalid argument],[bytes_written=0,to_write=51482,offset = 23111209], errno: 29, strerror: Illegal seek, e.what() = DB::Exception, Stack trace:\n\n\n       0x1d272d3\tStackTrace::StackTrace() [tiflash+30569171]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0x1d248d6\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30558422]\n                \tdbms/src/Common/Exception.h:41\n       0x1d302ba\tDB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+30606010]\n                \tdbms/src/Common/Exception.cpp:59\n       0x79d59d9\tvoid DB::PageUtil::writeFile<std::__1::shared_ptr<DB::WritableFile> >(std::__1::shared_ptr<DB::WritableFile>&, unsigned long, char*, unsigned long, std::__1::shared_ptr<DB::WriteLimiter> const&, bool, bool, bool) [tiflash+127752665]\n                \tdbms/src/Storages/Page/PageUtil.h:209\n       0x79f6925\tDB::PS::V2::PageFile::Writer::write(DB::WriteBatch&, DB::PS::V2::PageEntriesEdit&, std::__1::shared_ptr<DB::WriteLimiter> const&, bool) [tiflash+127887653]\n                \tdbms/src/Storages/Page/V2/PageFile.cpp:807\n       0x7a09d6e\tDB::PS::V2::PageStorage::writeImpl(DB::WriteBatch&&, std::__1::shared_ptr<DB::WriteLimiter> const&) [tiflash+127966574]\n                \tdbms/src/Storages/Page/V2/PageStorage.cpp:533\n       0x7a8d508\tDB::PageWriter::write(DB::WriteBatch&&, std::__1::shared_ptr<DB::WriteLimiter>) const [tiflash+128505096]\n                \tdbms/src/Storages/Page/PageStorage.cpp:460\n       0x77a86e6\tDB::DM::WriteBatches::writeLogAndData() [tiflash+125470438]\n                \tdbms/src/Storages/DeltaMerge/WriteBatches.h:115\n       0x78b94dc\tDB::DM::ColumnFileFlushTask::prepare(DB::DM::WriteBatches&) [tiflash+126588124]\n                \tdbms/src/Storages/DeltaMerge/Delta/ColumnFileFlushTask.cpp:58\n       0x78aac9b\tDB::DM::DeltaValueSpace::flush(DB::DM::DMContext&) [tiflash+126528667]\n                \tdbms/src/Storages/DeltaMerge/Delta/DeltaValueSpace.cpp:173\n       0x77c405a\tDB::DM::Segment::flushCache(DB::DM::DMContext&) [tiflash+125583450]\n                \tdbms/src/Storages/DeltaMerge/Segment.cpp:1328\n       0x77900a4\tDB::DM::DeltaMergeStore::flushCache(std::__1::shared_ptr<DB::DM::DMContext> const&, DB::DM::RowKeyRange const&) [tiflash+125370532]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:1008\n       0x1e6aa4f\tDB::DM::DeltaMergeStore::flushCache(DB::Context const&, DB::DM::RowKeyRange const&) [tiflash+31894095]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.h:374\n       0x78ffd7b\tDB::KVStore::tryFlushRegionCacheInStorage(DB::TMTContext&, DB::Region const&, Poco::Logger*) [tiflash+126877051]\n                \tdbms/src/Storages/Transaction/KVStore.cpp:153\n       0x790284a\tDB::KVStore::handleUselessAdminRaftCmd(raft_cmdpb::AdminCmdType, unsigned long, unsigned long, unsigned long, DB::TMTContext&) [tiflash+126888010]\n                \tdbms/src/Storages/Transaction/KVStore.cpp:383\n       0x7902a2a\tDB::KVStore::handleAdminRaftCmd(raft_cmdpb::AdminRequest&&, raft_cmdpb::AdminResponse&&, unsigned long, unsigned long, unsigned long, DB::TMTContext&) [tiflash+126888490]\n                \tdbms/src/Storages/Transaction/KVStore.cpp:410\n       0x791b925\tHandleAdminRaftCmd [tiflash+126990629]\n                \tdbms/src/Storages/Transaction/ProxyFFI.cpp:114\n  0x7fa0a816fbc0\traftstore::engine_store_ffi::_$LT$impl$u20$raftstore..engine_store_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::handle_admin_raft_cmd::hc999c7ac97c80db7 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+20614080]\n  0x7fa0a8910841\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::exec_admin_cmd::hc9d6a09e7d1209ef [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28612673]\n  0x7fa0a89042e9\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::process_raft_cmd::h4c8bba241d5ee4d4 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28562153]\n  0x7fa0a8902d6c\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::handle_raft_committed_entries::h478c4bad834ddec7 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28556652]\n  0x7fa0a8922f91\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_apply::he4c0b1137ae7cad9 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28688273]\n  0x7fa0a8925bfe\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_tasks::h5b138e5889c69e66 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28699646]\n  0x7fa0a8928510\t_$LT$raftstore..store..fsm..apply..ApplyPoller$LT$EK$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..apply..ApplyFsm$LT$EK$GT$$C$raftstore..store..fsm..apply..ControlFsm$GT$$GT$::handle_normal::he20c24833a37424f [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28710160]\n  0x7fa0a859b609\tbatch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::hec57674a22fdcc30 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+24987145]\n  0x7fa0a8c44cd8\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h189c0b2868fa537c [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+31972568]\n  0x7fa0a847cec1\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h7c53021c905641b6 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+23813825]\n  0x7fa0a8e5afca\tstd::sys::unix::thread::Thread::new::thread_start::hd39c5f08bdcda277 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+34160586]\n  0x7fa0a64a4ea5\tstart_thread [libpthread.so.0+32421]\n  0x7fa0a5fb796d\t__clone [libc.so.6+1042797]"] [thread_id=40]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

6.1

Start with dbms/src/Storages/Page/PageUtil.h and the write path in PageFile.cpp, PageStorage.cpp, and WriteBatches.h. Trace the Illegal seek during profiling on RHEL 8 through DeltaValueSpace.cpp and the listed KVStore entry points. Done means the reported profiling scenario no longer brings TiFlash down and the affected write completes successfully.