echo failed after fio test, seems cluster is unavailable forever

After building a 3fs cluster (1 meta, 2 storage), we mount /3fs/stage success, and take a fio test success too

But after fio test, we execute a command: echo "any" > any.txt, it stuck for a miniute and make an error:
bash: Input/output error
It seems that the 3fs cluster is unavailable forever:

If we rebuild cluster and do echo something before fio test, then cluster is unavailable immediately and do fio failed forever

**error log for storage 1:**

[2025-03-11T06:58:03.981553079+00:00 DefaultPool5: 706 TargetMap.cc:39 ERROR] chain ChainId(900100002) not found
[2025-03-11T06:58:03.981570917+00:00 DefaultPool5: 706 TargetMap.cc:49 ERROR] target TargetId(101000100102) not found
[2025-03-11T06:58:03.982536879+00:00 DefaultPool5: 706 TargetMap.cc:39 ERROR] chain ChainId(900100003) not found
[2025-03-11T06:58:03.982543139+00:00 DefaultPool5: 706 TargetMap.cc:49 ERROR] target TargetId(101000100103) not found
[2025-03-11T06:58:03.983514391+00:00 DefaultPool5: 706 TargetMap.cc:39 ERROR] chain ChainId(900100004) not found
[2025-03-11T06:58:03.983520950+00:00 DefaultPool5: 706 TargetMap.cc:49 ERROR] target TargetId(101000100104) not found
[2025-03-11T06:58:03.984267202+00:00 DefaultPool5: 706 TargetMap.cc:39 ERROR] chain ChainId(900100005) not found
[2025-03-11T06:58:03.984272122+00:00 DefaultPool5: 706 TargetMap.cc:49 ERROR] target TargetId(101000100105) not found
[2025-03-11T06:58:03.984910094+00:00 DefaultPool5: 706 TargetMap.cc:39 ERROR] chain ChainId(900100006) not found
[2025-03-11T06:58:03.984916183+00:00 DefaultPool5: 706 TargetMap.cc:49 ERROR] target TargetId(101000100106) not found
[2025-03-11T06:59:10.007730815+00:00 DefaultPool5: 706 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x5604cc228788, timeout: 100ms, peer address: RDMA://29.226.67.231:8010
[2025-03-11T06:59:10.007801418+00:00 DefaultPool5: 706 ReliableForwarding.cc:234 ERROR] forward timeout, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000000-F8000000-00000063"},"rdmabuf":{"addr":0,"size":0,"rkeys":[]},"updateVer":1,"updateType":"TRUNCATE","checksum":{"type":"NONE","value":0},"inlinebuf":"std::vector(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"41a6715e-647d-4901-a926-8a9a3b8c470d","hostname":"xxx"},"requestId":7,"channel":{"id":6,"seqnum":193}},"retryCount":0,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result RPC::Timeout(2005)
[2025-03-11T06:59:10.208153934+00:00 DefaultPool5: 706 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x5604cc228788, timeout: 200ms, peer address: RDMA://29.226.67.231:8010
[2025-03-11T06:59:10.208212459+00:00 DefaultPool5: 706 ReliableForwarding.cc:234 ERROR] forward timeout, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000000-F8000000-00000063"},"rdmabuf":{"addr":0,"size":0,"rkeys":[]},"updateVer":1,"updateType":"TRUNCATE","checksum":{"type":"NONE","value":0},"inlinebuf":"std::vector(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"41a6715e-647d-4901-a926-8a9a3b8c470d","hostname":"xxx"},"requestId":7,"channel":{"id":6,"seqnum":193}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result RPC::Timeout(2005)
[2025-03-11T06:59:10.608390497+00:00 DefaultPool5: 706 StorageMessenger.cc:31 ERROR] RPC communication error: RPC::Timeout(2005), request: 0x5604cc228788, timeout: 400ms, peer address: RDMA://29.226.67.231:8010
[2025-03-11T06:59:10.608440913+00:00 DefaultPool5: 706 ReliableForwarding.cc:234 ERROR] forward timeout, req {"payload":{"offset":0,"length":1048576,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100004,"chainVer":1},"chunkId":"00000000-00000000-F8000000-00000063"},"rdmabuf":{"addr":0,"size":0,"rkeys":[]},"updateVer":1,"updateType":"TRUNCATE","checksum":{"type":"NONE","value":0},"inlinebuf":"std::vector(0)"},"options":{"isSyncing":false,"fromClient":false,"commitChainVer":0},"tag":{"clientId":{"uuid":"41a6715e-647d-4901-a926-8a9a3b8c470d","hostname":"xxx"},"requestId":7,"channel":{"id":6,"seqnum":193}},"retryCount":2,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result RPC::Timeout(2005)
[2025-03-11T06:59:10.703535623+00:00 CliConn1: 655 IOWorker.cc:89 ERROR] error: RPC::SocketClosed(2013) IBSocket closed before connected
[2025-03-11T06:59:10.703539842+00:00 CliConn0: 654 IOWorker.cc:89 ERROR] error: RPC::SocketClosed(2013) IBSocket closed before connected
[2025-03-11T06:59:10.706422350+00:00 CliConn0: 654 IOWorker.cc:89 ERROR] error: RPC::SocketClosed(2013)
[2025-03-11T07:09:52.590311493+00:00 UpdatePool3: 688 StorageOperator.cc:432 CRITICAL] write update failed, req {"payload":{"offset":0,"length":3,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100005,"chainVer":1},"chunkId":"00000000-00000000-F9000000-00000000"},"rdmabuf":{"addr":140036542361328,"size":3,"rkeys":[{"rkey":2269858,"devId":0}]},"updateVer":0,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3827769471},"inlinebuf":"std::vector(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"41a6715e-647d-4901-a926-8a9a3b8c470d","hostname":"xxx"},"requestId":9125577,"channel":{"id":1,"seqnum":75009}},"retryCount":0,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{Storage::ChunkWriteFailed(4011) IoError("io error: Os { code: 22, kind: InvalidInput, message: \\"Invalid argument\\" }")} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(1)}

----------------------------------------
**error log for storage 2:**

[2025-03-11T06:58:03.982336564+00:00 DefaultPool1: 701 TargetMap.cc:39 ERROR] chain ChainId(900100002) not found
[2025-03-11T06:58:03.982351173+00:00 DefaultPool1: 701 TargetMap.cc:49 ERROR] target TargetId(101000200102) not found
[2025-03-11T06:58:03.983203572+00:00 DefaultPool1: 701 TargetMap.cc:39 ERROR] chain ChainId(900100003) not found
[2025-03-11T06:58:03.983211291+00:00 DefaultPool1: 701 TargetMap.cc:49 ERROR] target TargetId(101000200103) not found
[2025-03-11T06:58:03.984173059+00:00 DefaultPool1: 701 TargetMap.cc:39 ERROR] chain ChainId(900100004) not found
[2025-03-11T06:58:03.984180759+00:00 DefaultPool1: 701 TargetMap.cc:49 ERROR] target TargetId(101000200104) not found
[2025-03-11T06:58:03.984920978+00:00 DefaultPool1: 701 TargetMap.cc:39 ERROR] chain ChainId(900100005) not found
[2025-03-11T06:58:03.984927098+00:00 DefaultPool1: 701 TargetMap.cc:49 ERROR] target TargetId(101000200105) not found
[2025-03-11T06:58:03.985740281+00:00 DefaultPool1: 701 TargetMap.cc:39 ERROR] chain ChainId(900100006) not found
[2025-03-11T06:58:03.985745220+00:00 DefaultPool1: 701 TargetMap.cc:49 ERROR] target TargetId(101000200106) not found
[2025-03-11T06:59:25.062772549+00:00 SvrConn0: 605 IBSocket.cc:738 CRITICAL] IBSocket RDMA://xxx/mlx5_bond_1:1/8808 accept timeout, port {mlx5_bond_1:1, ifaddrs [29.137.34.211/24], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-03-11T06:59:25.062772549+00:00 SvrConn1: 606 IBSocket.cc:738 CRITICAL] IBSocket RDMA://xxx/mlx5_bond_1:1/8809 accept timeout, port {mlx5_bond_1:1, ifaddrs [29.137.34.211/24], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-03-11T06:59:25.062853862+00:00 SvrConn0: 605 Listener.cc:218 ERROR] IBSocket RDMA://xxx/mlx5_bond_1:1/8808 still in ACCEPTED state after wait 15s
[2025-03-11T06:59:25.062854112+00:00 SvrConn1: 606 Listener.cc:218 ERROR] IBSocket RDMA://xxx/mlx5_bond_1:1/8809 still in ACCEPTED state after wait 15s
[2025-03-11T07:10:02.590879406+00:00 UpdatePool7: 691 StorageOperator.cc:432 CRITICAL] write update failed, req {"payload":{"offset":0,"length":3,"chunkSize":1048576,"key":{"vChainId":{"chainId":900100005,"chainVer":2},"chunkId":"00000000-00000000-F9000000-00000000"},"rdmabuf":{"addr":140036542361328,"size":3,"rkeys":[{"rkey":2269858,"devId":0}]},"updateVer":0,"updateType":"WRITE","checksum":{"type":"CRC32C","value":3827769471},"inlinebuf":"std::vector(0)"},"options":{"isSyncing":false,"fromClient":true,"commitChainVer":0},"tag":{"clientId":{"uuid":"41a6715e-647d-4901-a926-8a9a3b8c470d","hostname":"xxx"},"requestId":9125577,"channel":{"id":1,"seqnum":75009}},"retryCount":1,"userInfo":"(uid 0, gid 0, groups ())","featureFlags":0,"debugFlags":{"injectRandomServerError":false,"injectRandomClientError":false,"numOfInjectPtsBeforeFail":0}}, result length{Storage::ChunkWriteFailed(4011) IoError("io error: Os { code: 22, kind: InvalidInput, message: \\"Invalid argument\\" }")} version{ChunkVer(0)/ChunkVer(0)} checksum{CRC32C#00000000} {ChainVersion(2)}