mgmtd_main and admin_cli core when deploy with E810 RDMA card

1. we try to deploy a 3FS cluster with Intel E810 RDMA card;
2. the admin_cli exit abnormally when running "/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.100.100.112:8000"]' "list-nodes"
" command after we have start mgmtd_main service successfully;
3. after a while the mgmtd_main also exit abnormally.

It seems that the two abnormal exit due to same reason, just happen when trying to close a IBSocket.

A. ###########the core stack of admin_cli
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.100.100.112:8000"]' "list-nodes"
Id Type Status Hostname Pid Tags LastHeartbeatTime ConfigVersion ReleaseVersion
1 MGMTD PRIMARY_MGMTD spr-9 374052 [] N/A 1(UPTODATE) 250228-dev-1-999999-3a30c53b
100 META HEARTBEAT_CONNECTING spr-9 0 [] N/A 0 0-dev-0-0-00000000
*** Aborted at 1742303109 (Unix time, try 'date -d @1742303109') ***
*** Signal 11 (SIGSEGV) (0x0) received by PID 374105 (pthread TID 0x7f7c6702ed40) (linux TID 374105) (code: address not mapped to object), stack trace: ***
@ 000000000245250f folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
/home/3fs/3fs/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449
@ 000000000004251f (unknown)
@ 0000000000006cf0 (unknown)
@ 00000000022a373f hf3fs::net::IBSocket::closeGracefully()
/usr/include/infiniband/verbs.h:3327
-> /home/3fs/3fs/src/common/net/ib/IBSocket.cc
@ 00000000022a50c2 hf3fs::net::IBSocket::Drainer::create(std::unique_ptr >, std::weak_ptr)
/home/3fs/3fs/src/common/net/ib/IBSocket.cc:1094
@ 00000000022a6968 hf3fs::net::IBSocketManager::close(std::unique_ptr >)
/home/3fs/3fs/src/common/net/ib/IBSocket.cc:1203
@ 00000000022915f3 hf3fs::net::IBManager::close(std::unique_ptr >)
/home/3fs/3fs/src/common/net/ib/IBDevice.cc:757
@ 0000000002255746 hf3fs::net::Transport::~Transport()
/home/3fs/3fs/src/common/net/Transport.cc:64
@ 000000000367d539 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr_base.h:175
-> /home/3fs/3fs/src/client/bin/admin_cli.cc
@ 00000000037460a6 std::_Rb_tree, std::pair const, unsigned int>, std::_Select1st const, unsigned int> >, std::less >, std::allocator const, unsigned int> > >::_M_erase(std::_Rb_tree_node const, unsigned int> >*)
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr_base.h:353
-> /home/3fs/3fs/src/client/bin/admin_cli.cc
@ 00000000022605bd hf3fs::net::TransportPool::dropConnections(bool, bool)
/usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_tree.h:986
-> /home/3fs/3fs/src/common/net/TransportPool.cc
@ 00000000022315f6 hf3fs::net::IOWorker::stopAndJoin()
/home/3fs/3fs/src/common/net/IOWorker.h:76
-> /home/3fs/3fs/src/common/net/IOWorker.cc
@ 00000000036649c3 main
/home/3fs/3fs/src/common/net/Client.h:51
-> /home/3fs/3fs/src/client/bin/admin_cli.cc
@ 0000000000029d8f (unknown)
@ 0000000000029e3f __libc_start_main
@ 0000000001a03ca4 _start

B. ########## the core stack of mgmtd_main
mgmtd_main[374052]: *** Aborted at 1742303146 (Unix time, try 'date -d @1742303146') ***
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** Signal 11 (SIGSEGV) (0x0) received by PID 374052 (pthread TID 0x7efeb4ff9640) (linux TID 374084) (code: address not mapped to object), stack trace: ***
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000007445ef folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/experimental/symbolizer/SignalHandler.cpp:449
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 000000000004251f (unknown)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000006cf0 (unknown)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000005a0588 hf3fs::net::IBSocket::check()
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /usr/include/infiniband/verbs.h:3327
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/common/net/ib/IBSocket.cc
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000005589c9 hf3fs::net::TransportSet::checkAll(hf3fs::Duration)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/src/common/net/Transport.h:64
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/common/net/TransportPool.cc
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000559e51 hf3fs::net::TransportPool::checkConnections(hf3fs::net::Address, hf3fs::Duration)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/src/common/net/TransportPool.cc:198
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000536454 hf3fs::net::ServiceGroup::checkConnectionsRegularly() [clone .resume]
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/src/common/net/IOWorker.h:82
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/common/net/ServiceGroup.cc
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000007d0183 folly::resumeCoroutineWithNewAsyncStackRoot(std::__n4861::coroutine_handle, folly::AsyncStackFrame&)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/coroutine:135
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/third_party/folly/folly/tracing/AsyncStack.cpp
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000227498 folly::coro::detail::ViaCoroutinePromiseBase::executeContinuation()
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/experimental/coro/ViaIfAsync.h:90
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/src/mgmtd/MgmtdConfigFetcher.cc
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 000000000070f52a folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/Function.h:375
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/third_party/folly/folly/executors/ThreadPoolExecutor.cpp
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000006fa5aa folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/executors/CPUThreadPoolExecutor.cpp:333
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000711edf void std::__invoke_impl), folly::ThreadPo>
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/invoke.h:74
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: -> /home/3fs/3fs/third_party/folly/folly/executors/ThreadPoolExecutor.cpp
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000000dc252 (unknown)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 0000000000094ac2 (unknown)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 000000000012684f (unknown)
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** Check failure async stack trace: ***
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** First async stack root: 0x7efeb4bdd988, normal stack frame pointer holding async stack root: 0x7efeb4bdd9c0, return address: 0x561e113b8499 ***
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: *** First async stack frame pointer: 0x561e5195c018, return address: 0x561e116929e7, async stack trace: ***
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: @ 00000000005019e7 folly::coro::detail::InlineTaskDetached folly::coro::TaskWithExecutor::startImpl::start() &&:>
Mar 18 21:05:46 spr-9 mgmtd_main[374052]: /home/3fs/3fs/third_party/folly/folly/experimental/coro/ViaIfAsync.h:0

C. ######### the ibv_devinfo of E810 RDMA card
hca_id: irdma0
transport: InfiniBand (0)
fw_ver: 1.74
node_guid: 6efe:54ff:fe40:c0d0
sys_image_guid: 6efe:54ff:fe40:c0d0
vendor_id: 0x8086
vendor_part_id: 5522
hw_ver: 0x2
phys_port_cnt: 1
max_mr_size: 0x200000000000
page_size_cap: 0x40201000
max_qp: 4092
max_qp_wr: 8127
device_cap_flags: 0x01221000
RC_RNR_NAK_GEN
MEM_WINDOW
MEM_MGT_EXTENSIONS
MEM_WINDOW_TYPE_2B
max_sge: 6
max_sge_rd: 6
max_cq: 8189
max_cqe: 1048574
max_mr: 4194301
max_pd: 262141
max_qp_rd_atom: 8
max_ee_rd_atom: 0
max_res_rd_atom: 0
max_qp_init_rd_atom: 8
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_NONE (0)
max_ee: 0
max_rdd: 0
max_mw: 0
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 4096
max_mcast_qp_attach: 8
max_total_mcast_qp_attach: 32768
max_ah: 131072
max_fmr: 0
max_srq: 0
max_pkeys: 1
local_ca_ack_delay: 0
general_odp_caps:
rc_odp_caps:
NO SUPPORT
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
NO SUPPORT
xrc_odp_caps:
NO SUPPORT
completion_timestamp_mask not supported
core clock not supported
device_cap_flags_ex: 0x1221000
tso_caps:
max_tso: 0
rss_caps:
max_rwq_indirection_tables: 0
max_rwq_indirection_table_size: 0
rx_hash_function: 0x0
rx_hash_fields_mask: 0x0
max_wq_type_rq: 0
packet_pacing_caps:
qp_rate_limit_min: 0kbps
qp_rate_limit_max: 0kbps
tag matching not supported
num_comp_vectors: 65
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 1
port_lmc: 0x00
link_layer: Ethernet
max_msg_sz: 0x7fffffff
port_cap_flags: 0x04050000
port_cap_flags2: 0x0000
max_vl_num: invalid value (0)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 1
gid_tbl_len: 32
subnet_timeout: 0
init_type_reply: 0
active_width: 4X (2)
active_speed: 25.0 Gbps (32)
phys_state: LINK_UP (5)
GID[ 0]: fe80::6efe:54ff:fe40:c0d0, RoCE v2
GID[ 1]: ::ffff:192.100.100.112, RoCE v2

D. ####### and we have change the max_sge = 1 and max_rd_atomic = 8 according to the capacity of irdma0 device

E. we can run ib_write_bw with the RDMA device successfully

Actually, there are _opt_3fs_bin_admin_cli.0.crash and _opt_3fs_bin_mgmtd_main.0.crash under /var/crash directory, but we are not able to analyze them, as they not a type of file known by gdb.