rbd-target-api stuck on uninterruptible sleep

Environment:
ceph-iscsi, rtslib-fb, configshell-fb and tcmu-runner all checked out from master branches, following: https://docs.ceph.com/docs/master/rbd/iscsi-target-cli-manual-install/
Ubuntu 19.10

I am aware of similar a similar issue, but I don't know if they refer to my case:
https://github.com/ceph/ceph-iscsi-cli/issues/102

Testing failover scenario with 2 gateways: one target instance goes down and initiators running multipath should choose the other available. To simulate the failover, I (accidentally) ran:
sudo systemctl stop tcmu-runner
sudo systemctl stop rbd-target-gw
sudo systemctl stop rbd-target-api

By running this sequence, I got rbd-target-api stuck on D state.
Checking rbd-* process with ps, I can see:

`$ sudo ps aux | grep rbd
root 2685 0.0 5.8 1121860 119116 ? Ds Dec28 1:32 /usr/bin/python3 /usr/bin/rbd-target-api
ubuntu 19379 0.0 0.0 6428 852 pts/0 S+ 22:01 0:00 grep --color=auto rbd
`
`$ sudo ps -flp 2685
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
4 D root 2685 1 0 80 0 - 280465 iscsit Dec28 ? 00:01:32 /usr/bin/python3 /usr/bin/rbd-target-api
`

tcmu-runner is not present on ps.

Digging into process 2685 (rbd-target-api), I can see following stacktrace, memory map and logs:
rbd-target-api logs (from shutdown received onwards): https://pastebin.ubuntu.com/p/hMVZDxTVNT/
/proc/2685/stack: https://pastebin.ubuntu.com/p/G4f5B5BWd6/
/proc/2685/maps: https://pastebin.ubuntu.com/p/Pb56KqyJCj/

Stacktrace shows that rbd-target-api is on uninterruptible sleep waiting on ISCSI to finish. However, tcmu-runner is already dead so it will never return.

I understand rbd-target-api needs to run some iscsi-related tasks, for example, removing itself as a target from gateway list. However, it should first ensure tcmu-runner is healthy before running any action that may cause it to hang. Otherwise, if tcmu-runner is not running, it should raise an exception and fail; and giving some feedback to the operator.

Also, I saw gwcli command was hanging and not only on the gateway that failed.

I can imagine this scenario happening in operations where tcmu-runner crashes for some unrelated reason and rbd-target-api eventually hangs waiting for it to reply. The only solution I see is to reboot the node.

Start with the rbd-target-api shutdown path and the tcmu-runner/rbd-target-gw stop sequence, then inspect the linked logs and /proc/2685/stack. Check how the API behaves when tcmu-runner has stopped and whether gwcli is affected on other gateways. Done means the API does not remain in uninterruptible sleep and gives the operator useful failure feedback.

Written by the indexing model from the issue text.