After deploying mgmtd and meta, the list-nodes command occasionally works and occasionally fails. What could be the reason?

After deploying mgmtd and meta, the list-nodes command occasionally works and occasionally fails. What could be the reason?

The error message is as follows:
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://172.31.12.114:8000"]' "list-nodes"
Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD testnode1ssd 4621 [] N/A 1(UPTODATE) 250228-dev-1-999999-fcef04fa
100 META HEARTBEAT_CONNECTED testnode1ssd 440 [] 2025-04-23 09:39:31 1(UPTODATE) 250228-dev-1-999999-fcef04fa
root@testnode1ssd:/opt/3fs/etc# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://172.31.12.114:8000"]' "list-nodes"
Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD testnode1ssd 4621 [] N/A 1(UPTODATE) 250228-dev-1-999999-fcef04fa
100 META HEARTBEAT_CONNECTED testnode1ssd 440 [] 2025-04-23 09:43:01 1(UPTODATE) 250228-dev-1-999999-fcef04fa
root@testnode1ssd:/opt/3fs/etc# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://172.31.12.114:8000"]' "list-nodes"
Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD testnode1ssd 4621 [] N/A 1(UPTODATE) 250228-dev-1-999999-fcef04fa
100 META HEARTBEAT_CONNECTED testnode1ssd 440 [] 2025-04-23 09:43:01 1(UPTODATE) 250228-dev-1-999999-fcef04fa
root@testnode1ssd:/opt/3fs/etc# /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://172.31.12.114:8000"]' "list-nodes"
Encounter error: 6004(MgmtdClient::RoutingInfoNotReady)

mgmtd error log:
[2025-04-23T09:35:26.145277402+00:00 SvrProc0: 4740 IBConnect.cc:681 ERROR] IBSocket RDMA://testnode1ssd/mlx5_bond_0:1/8500 failed to modify QP to RTR, errno 110, peer {"hostname":"testnode1ssd","dev":2,"dev_name":"mlx5_bond_0","port":1,"mtu":3,"qp_num":8500,"linklayer":{"type":"IBConnectRoCEInfo","value":{"gid":"\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u001f\\fr"}}}, config {"sl":0,"traffic_class":0,"gid_index":0,"pkey_index":0,"start_psn":0,"min_rnr_timer":1,"timeout":14,"retry_cnt":7,"rnr_retry":0,,
"max_sge":16,"max_rdma_wr":128,"max_rdma_wr_per_post":32,"max_rd_atomic":16,"buf_size":16384,"send_buf_cnt":32,"buf_ack_batch":8,"buf_signal_batch":8,"event_ack_batch":128,"record_latency_per_peer":false}
[2025-04-23T09:35:26.145339871+00:00 SvrProc0: 4740 IBConnect.cc:168 ERROR] IBConnectService::connect failed to accept from 172.31.12.114, error RPC::ConnectFailed(2014) IBSocket failed to modify QP to RTR.
[2025-04-23T09:35:36.642292569+00:00 SvrConn1: 4747 IBSocket.cc:738 CRITICAL] IBSocket RDMA://testnode1ssd/mlx5_2:1/2059 accept timeout, port {mlx5_bond_0:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-04-23T09:35:36.642351209+00:00 SvrConn1: 4747 Listener.cc:218 ERROR] IBSocket RDMA://testnode1ssd/mlx5_2:1/2059 still in ACCEPTED state after wait 15s
[2025-04-23T09:35:37.742451498+00:00 SvrConn0: 4746 IBSocket.cc:738 CRITICAL] IBSocket RDMA://testnode1ssd/mlx5_3:1/4342 accept timeout, port {mlx5_bond_0:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-04-23T09:35:37.742510895+00:00 SvrConn0: 4746 Listener.cc:218 ERROR] IBSocket RDMA://testnode1ssd/mlx5_3:1/4342 still in ACCEPTED state after wait 15s
[2025-04-23T09:35:42.484212045+00:00 SvrConn0: 4746 IBSocket.cc:738 CRITICAL] IBSocket RDMA://testnode1ssd/mlx5_2:1/2062 accept timeout, port {mlx5_bond_0:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-04-23T09:35:42.484264204+00:00 SvrConn0: 4746 Listener.cc:218 ERROR] IBSocket RDMA://testnode1ssd/mlx5_2:1/2062 still in ACCEPTED state after wait 15s
[2025-04-23T09:35:43.582416723+00:00 SvrConn1: 4747 IBSocket.cc:738 CRITICAL] IBSocket RDMA://testnode1ssd/mlx5_3:1/4344 accept timeout, port {mlx5_bond_0:1, ifaddrs [], zones [UNKNOWN], ETHERNET, ACTIVE}
[2025-04-23T09:35:43.582465859+00:00 SvrConn1: 4747 Listener.cc:218 ERROR] IBSocket RDMA://testnode1ssd/mlx5_3:1/4344 still in ACCEPTED state after wait 15s

Could it be due to network timeout? However, the network latency appears normal upon inspection