Network issues possibly causing ceph rbd cmd timeouts

hi can anyone tell me what the f is going on here?
sorry for multiple questions in a row but I'm really tired of this

i was using ceph-iscsi with multipath and got errors after googling found out about multipath
i reconfigured everything and now i have one esxi and one gateway
3 luns

esxi version is 6.5u3

the following is my system info and logs
I should mention my iscsi server is hpdl380 g7

I'm struggling with the same kind of behavior for over a mount and so much reconfiguring and tweaking I'm really really tired
**do i have to drop ceph because of VMware compatibility**

**does anyone successfully used ceph for VMWare in productional environment?**

uname -a
Linux ceph-client02 3.10.0-1062.4.3.el7.x86_64 #1 SMP Wed Nov 13 23:58:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

lspci
00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:02.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:04.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 4 (rev 13)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 13)
00:06.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 6 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:08.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 8 (rev 13)
00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
00:0a.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13)
00:0d.0 Host bridge: Intel Corporation Device 343a (rev 13)
00:0d.1 Host bridge: Intel Corporation Device 343b (rev 13)
00:0d.2 Host bridge: Intel Corporation Device 343c (rev 13)
00:0d.3 Host bridge: Intel Corporation Device 343d (rev 13)
00:0d.4 Host bridge: Intel Corporation 7500/5520/5500/X58 Physical Layer Port 0 (rev 13)
00:0d.5 Host bridge: Intel Corporation 7500/5520/5500 Physical Layer Port 1 (rev 13)
00:0d.6 Host bridge: Intel Corporation Device 341a (rev 13)
00:0e.0 Host bridge: Intel Corporation Device 341c (rev 13)
00:0e.1 Host bridge: Intel Corporation Device 341d (rev 13)
00:0e.2 Host bridge: Intel Corporation Device 341e (rev 13)
00:0e.3 Host bridge: Intel Corporation Device 341f (rev 13)
00:0e.4 Host bridge: Intel Corporation Device 3439 (rev 13)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.2 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 3
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.3 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02)
02:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 04)
02:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 04)
02:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 01)
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
04:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
04:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
05:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)
0b:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
0b:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
0b:00.2 Fibre Channel: Emulex Corporation OneConnect 10Gb FCoE Initiator (be3) (rev 01)
0b:00.3 Fibre Channel: Emulex Corporation OneConnect 10Gb FCoE Initiator (be3) (rev 01)
3e:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)
3e:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)
3e:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)
3e:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)
3e:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)
3e:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)
3e:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)
3e:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)
3e:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)
3e:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)
3e:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)
3e:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)
3e:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)
3e:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)
3e:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)
3e:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)
3e:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)
3e:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)
3e:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)
3e:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)
3e:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)
3e:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)
3e:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)
3e:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)
3f:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)
3f:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)
3f:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)
3f:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)
3f:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)
3f:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)
3f:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)
3f:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)
3f:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)
3f:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)
3f:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)
3f:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)
3f:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)
3f:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)
3f:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)
3f:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)
3f:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)
3f:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)
3f:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)
3f:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)
3f:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)
3f:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)
3f:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)
3f:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)

bond0: flags=5187 mtu 9000
inet x.x.x.x netmask 255.255.255.0 broadcast x.x.x.x
inet6 fe80::82f1:ba05:ad23:a878 prefixlen 64 scopeid 0x20
ether x:x:x:x:x txqueuelen 1000 (Ethernet)
RX packets 19094018 bytes 85766194981 (79.8 GiB)
RX errors 0 dropped 34014 overruns 1794 frame 0
TX packets 20622318 bytes 86347023115 (80.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.cdm-03: Handler connection lost (lock state 1)
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Nov 25 11:28:42 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Nov 25 11:28:46 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.cdm-03: Starting lock acquisition operation.
Nov 25 11:28:46 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.cdm-03: Acquired exclusive lock.
Nov 25 11:28:46 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.cdm-03: Lock acquisition successful
Nov 25 11:29:11 ceph-client02 kernel: ABORT_TASK: Found referenced iSCSI task_tag: 4909859
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.cdm-03: Handler connection lost (lock state 1)
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 4909859
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Nov 25 11:29:16 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Nov 25 11:29:20 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.cdm-03: Starting lock acquisition operation.
Nov 25 11:29:20 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.cdm-03: Acquired exclusive lock.
Nov 25 11:29:20 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.cdm-03: Lock acquisition successful
Nov 25 11:29:50 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:29:50 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.cdm-03: Handler connection lost (lock state 1)
Nov 25 11:29:50 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Nov 25 11:29:50 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Nov 25 11:29:54 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.cdm-03: Starting lock acquisition operation.
Nov 25 11:29:54 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.cdm-03: Acquired exclusive lock.
Nov 25 11:29:54 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.cdm-03: Lock acquisition successful
Nov 25 11:30:12 ceph-client02 kernel: ABORT_TASK: Found referenced iSCSI task_tag: 4910855
Nov 25 11:30:24 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:30:24 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.cdm-03: Handler connection lost (lock state 1)
Nov 25 11:30:24 ceph-client02 kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 4910855
Nov 25 11:30:24 ceph-client02 kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 4910855
Nov 25 11:30:24 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Nov 25 11:30:24 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Nov 25 11:30:28 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.cdm-03: Starting lock acquisition operation.
Nov 25 11:30:28 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.cdm-03: Acquired exclusive lock.
Nov 25 11:30:28 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.cdm-03: Lock acquisition successful
Nov 25 11:30:58 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:30:58 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.cdm-03: Handler connection lost (lock state 1)
Nov 25 11:30:58 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:30:58 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:30:58 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:30:58 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:30:58 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:30:58 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:30:58 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.
Nov 25 11:30:58 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Nov 25 11:30:58 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Nov 25 11:31:02 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.cdm-03: Starting lock acquisition operation.
Nov 25 11:31:02 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.cdm-03: Acquired exclusive lock.
Nov 25 11:31:02 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.cdm-03: Lock acquisition successful
Nov 25 11:31:04 ceph-client02 kernel: ABORT_TASK: Found referenced iSCSI task_tag: 4913273
Nov 25 11:31:12 ceph-client02 kernel: INFO: task jbd2/rbd0-8:30759 blocked for more than 120 seconds.
Nov 25 11:31:12 ceph-client02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 25 11:31:12 ceph-client02 kernel: jbd2/rbd0-8 D ffffa0ec6a3fd230 0 30759 2 0x00000080
Nov 25 11:31:12 ceph-client02 kernel: Call Trace:
Nov 25 11:31:12 ceph-client02 kernel: [] ? task_rq_unlock+0x20/0x20
Nov 25 11:31:12 ceph-client02 kernel: [] schedule+0x29/0x70
Nov 25 11:31:12 ceph-client02 kernel: [] jbd2_journal_commit_transaction+0x23c/0x19f0 [jbd2]
Nov 25 11:31:12 ceph-client02 kernel: [] ? account_entity_dequeue+0xae/0xd0
Nov 25 11:31:12 ceph-client02 kernel: [] ? dequeue_task_fair+0x41e/0x660
Nov 25 11:31:12 ceph-client02 kernel: [] ? __switch_to+0xce/0x580
Nov 25 11:31:12 ceph-client02 kernel: [] ? wake_up_atomic_t+0x30/0x30
Nov 25 11:31:12 ceph-client02 kernel: [] ? __schedule+0x448/0x9c0
Nov 25 11:31:12 ceph-client02 kernel: [] ? try_to_del_timer_sync+0x5e/0x90
Nov 25 11:31:12 ceph-client02 kernel: [] kjournald2+0xc9/0x260 [jbd2]
Nov 25 11:31:12 ceph-client02 kernel: [] ? wake_up_atomic_t+0x30/0x30
Nov 25 11:31:12 ceph-client02 kernel: [] ? commit_timeout+0x10/0x10 [jbd2]
Nov 25 11:31:12 ceph-client02 kernel: [] kthread+0xd1/0xe0
Nov 25 11:31:12 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Nov 25 11:31:12 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Nov 25 11:31:12 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Nov 25 11:31:12 ceph-client02 kernel: INFO: task kworker/u65:0:30791 blocked for more than 120 seconds.
Nov 25 11:31:12 ceph-client02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 25 11:31:12 ceph-client02 kernel: kworker/u65:0 D ffffa0f84a37c1c0 0 30791 2 0x00000080
Nov 25 11:31:12 ceph-client02 kernel: Workqueue: writeback bdi_writeback_workfn (flush-252:0)
Nov 25 11:31:12 ceph-client02 kernel: Call Trace:
Nov 25 11:31:12 ceph-client02 kernel: [] schedule+0x29/0x70
Nov 25 11:31:12 ceph-client02 kernel: [] wait_transaction_locked+0x85/0xd0 [jbd2]
Nov 25 11:31:12 ceph-client02 kernel: [] ? wake_up_atomic_t+0x30/0x30
Nov 25 11:31:12 ceph-client02 kernel: [] add_transaction_credits+0x268/0x2f0 [jbd2]
Nov 25 11:31:12 ceph-client02 kernel: [] ? ___slab_alloc+0x3ac/0x4f0
Nov 25 11:31:12 ceph-client02 kernel: [] start_this_handle+0x1a1/0x430 [jbd2]
Nov 25 11:31:12 ceph-client02 kernel: [] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 25 11:31:12 ceph-client02 kernel: [] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Nov 25 11:31:12 ceph-client02 kernel: [] ? ext4_writepages+0x43c/0xcf0 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] __ext4_journal_start_sb+0x69/0xe0 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] ext4_writepages+0x43c/0xcf0 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] ? generic_writepages+0x58/0x80
Nov 25 11:31:12 ceph-client02 kernel: [] do_writepages+0x21/0x50
Nov 25 11:31:12 ceph-client02 kernel: [] __writeback_single_inode+0x40/0x260
Nov 25 11:31:12 ceph-client02 kernel: [] ? wake_up_bit+0x25/0x30
Nov 25 11:31:12 ceph-client02 kernel: [] writeback_sb_inodes+0x1c4/0x430
Nov 25 11:31:12 ceph-client02 kernel: [] __writeback_inodes_wb+0x9f/0xd0
Nov 25 11:31:12 ceph-client02 kernel: [] wb_writeback+0x263/0x2f0
Nov 25 11:31:12 ceph-client02 kernel: [] ? get_nr_inodes+0x4c/0x70
Nov 25 11:31:12 ceph-client02 kernel: [] bdi_writeback_workfn+0x2cb/0x460
Nov 25 11:31:12 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Nov 25 11:31:12 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Nov 25 11:31:12 ceph-client02 kernel: [] ? manage_workers.isra.26+0x2a0/0x2a0
Nov 25 11:31:12 ceph-client02 kernel: [] kthread+0xd1/0xe0
Nov 25 11:31:12 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Nov 25 11:31:12 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Nov 25 11:31:12 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Nov 25 11:31:12 ceph-client02 kernel: INFO: task fio:31269 blocked for more than 120 seconds.
Nov 25 11:31:12 ceph-client02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 25 11:31:12 ceph-client02 kernel: fio D ffffa0f8690ce2a0 0 31269 1 0x00000084
Nov 25 11:31:12 ceph-client02 kernel: Call Trace:
Nov 25 11:31:12 ceph-client02 kernel: [] schedule+0x29/0x70
Nov 25 11:31:12 ceph-client02 kernel: [] schedule_timeout+0x221/0x2d0
Nov 25 11:31:12 ceph-client02 kernel: [] ? blk_flush_plug_list+0xce/0x230
Nov 25 11:31:12 ceph-client02 kernel: [] io_schedule_timeout+0xad/0x130
Nov 25 11:31:12 ceph-client02 kernel: [] wait_for_completion_io+0xfd/0x140
Nov 25 11:31:12 ceph-client02 kernel: [] ? wake_up_state+0x20/0x20
Nov 25 11:31:12 ceph-client02 kernel: [] blkdev_issue_zeroout+0x270/0x280
Nov 25 11:31:12 ceph-client02 kernel: [] ext4_issue_zeroout+0x32/0x40 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] ext4_ext_zeroout+0x2f/0x40 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] ext4_ext_handle_unwritten_extents+0xa0b/0xc00 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] ext4_ext_map_blocks+0x8cf/0xf60 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] ext4_map_blocks+0x155/0x6e0 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 25 11:31:12 ceph-client02 kernel: [] ? alloc_buffer_head+0x21/0x60
Nov 25 11:31:12 ceph-client02 kernel: [] _ext4_get_block+0x1df/0x220 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] ext4_get_block+0x16/0x20 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] __block_write_begin_int+0x198/0x650
Nov 25 11:31:12 ceph-client02 kernel: [] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 25 11:31:12 ceph-client02 kernel: [] ? _ext4_get_block+0x220/0x220 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] ? ext4_write_begin+0x116/0x440 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] __block_write_begin+0x11/0x20
Nov 25 11:31:12 ceph-client02 kernel: [] ext4_write_begin+0x18f/0x440 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] ext4_da_write_begin+0x2ae/0x360 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] generic_file_buffered_write+0x10f/0x270
Nov 25 11:31:12 ceph-client02 kernel: [] __generic_file_aio_write+0x1e2/0x400
Nov 25 11:31:12 ceph-client02 kernel: [] generic_file_aio_write+0x59/0xa0
Nov 25 11:31:12 ceph-client02 kernel: [] ext4_file_write+0xd2/0x1e0 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] ? __sb_start_write+0x58/0x120
Nov 25 11:31:12 ceph-client02 kernel: [] ? security_file_permission+0x27/0xa0
Nov 25 11:31:12 ceph-client02 kernel: [] ? ext4_write_checks.isra.8+0x150/0x150 [ext4]
Nov 25 11:31:12 ceph-client02 kernel: [] do_io_submit+0x3e3/0x8a0
Nov 25 11:31:12 ceph-client02 kernel: [] SyS_io_submit+0x10/0x20
Nov 25 11:31:12 ceph-client02 kernel: [] system_call_fastpath+0x25/0x2a
Nov 25 11:31:12 ceph-client02 kernel: [] ? system_call_after_swapgs+0xae/0x146
Nov 25 11:31:28 ceph-client02 kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 4913273
Nov 25 11:31:28 ceph-client02 kernel: ABORT_TASK: Found referenced iSCSI task_tag: 4913274
Nov 25 11:31:28 ceph-client02 kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 4913274
Nov 25 11:31:28 ceph-client02 kernel: ABORT_TASK: Found referenced iSCSI task_tag: 4913275
Nov 25 11:31:28 ceph-client02 kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 4913275
Nov 25 11:31:28 ceph-client02 kernel: ABORT_TASK: Found referenced iSCSI task_tag: 4913279
Nov 25 11:31:28 ceph-client02 kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 4913279
Nov 25 11:31:30 ceph-client02 kernel: Unable to locate ITT: 0x004af879 on CID: 0
Nov 25 11:31:30 ceph-client02 kernel: Unable to locate RefTaskTag: 0x004af879 on CID: 0.
Nov 25 11:31:30 ceph-client02 kernel: Unable to locate ITT: 0x004af87a on CID: 0
Nov 25 11:31:30 ceph-client02 kernel: Unable to locate RefTaskTag: 0x004af87a on CID: 0.

Start with the tcmu-runner timeout and connection-loss log lines, then inspect the bond0 RX drops and overruns alongside the Ceph iSCSI, multipath, and VMware configuration described. Done would require a reproducible failure, confirmed scope, and a maintainer-approved fix or documented compatibility finding.

Written by the indexing model from the issue text.