rtslib_fb: `db_root: cannot be changed because it's in use`

Hi,

i'm running ceph 17.2.6 via docker containers on el 8.8 and since the latest updates i get lots of kernel errors

`kernel: db_root: cannot be changed because it's in use`

and the target-api container is constantly rebooting:

```
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: Could not load gateway: Cannot set dbroot to /var/target. Please check if this directory exists.
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: Removing iSCSI target from LIO
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: No available network portal for target with iqn of 'iqn.2001-07.com.ceph:1691062051543'
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: (Gateway.load_config) successfully loaded existing target definition
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: lio disk deletion failed Cannot set dbroot to /var/target. Please check if this directory exists.
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: lio disk deletion failed Cannot set dbroot to /var/target. Please check if this directory exists.
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: (Gateway.load_config) successfully loaded existing target definition
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: OSError: [Errno 22] Invalid argument
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: During handling of the above exception, another exception occurred:
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: Traceback (most recent call last):
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 171, in _set_dbroot
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: fwrite(dbroot_path, self._default_dbroot+"\n")
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/rtslib_fb/utils.py", line 79, in fwrite
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: file_fd.write(str(string))
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: OSError: [Errno 22] Invalid argument
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: During handling of the above exception, another exception occurred:
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: Traceback (most recent call last):
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/bin/rbd-target-api", line 2930, in main
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: ceph_gw.define()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/gateway.py", line 267, in define
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: self.define_targets()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/gateway.py", line 240, in define_targets
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: target = self.define_target(iqn, target.get('ip_list', {}))
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/gateway.py", line 214, in define_target
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: GWClient.define_clients(self.logger, self.config, target.iqn)
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/client.py", line 318, in define_clients
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: client.manage('present') # ensure the client exists
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/client.py", line 609, in manage
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: client_exists = self.exists()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/client.py", line 519, in exists
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: r = lio_root.RTSRoot()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 88, in __init__
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: self._set_dbroot()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 175, in _set_dbroot
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: .format(self._default_dbroot))
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: rtslib_fb.utils.RTSLibError: Cannot set dbroot to /var/target. Please check if this directory exists.
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: During handling of the above exception, another exception occurred:
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: OSError: [Errno 22] Invalid argument
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: During handling of the above exception, another exception occurred:
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: Traceback (most recent call last):
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 171, in _set_dbroot
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: fwrite(dbroot_path, self._default_dbroot+"\n")
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/rtslib_fb/utils.py", line 79, in fwrite
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: file_fd.write(str(string))
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: OSError: [Errno 22] Invalid argument
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: During handling of the above exception, another exception occurred:
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: Traceback (most recent call last):
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/bin/rbd-target-api", line 3022, in
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: main()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/bin/rbd-target-api", line 2934, in main
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: ceph_gw.delete()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/gateway.py", line 351, in delete
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: lio = LIO()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/lio.py", line 10, in __init__
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: self.lio_root = root.RTSRoot()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 88, in __init__
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: self._set_dbroot()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 175, in _set_dbroot
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: .format(self._default_dbroot))
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: rtslib_fb.utils.RTSLibError: Cannot set dbroot to /var/target. Please check if this directory exists.
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: teardown: managing teardown after SIGCHLD
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: teardown: Waiting PID 80 to terminate
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: teardown: Process 80 is terminated
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: teardown: Bye Bye, container will die with return code 0
```

It seems there was already a fix provided which didn't make it into the ceph container:

https://bugzilla.redhat.com/show_bug.cgi?id=1676669
https://github.com/open-iscsi/rtslib-fb/pull/156

Infos:
```
el 8.8 / 4.18.0-477.21.1.el8_8.x86_64
ceph 17.2.6
docker setup
```

Regards,
Kilian

Start with rtslib_fb/root.py at _set_dbroot and the ceph_iscsi_config gateway.py and lio.py call sites shown in the traceback. Compare the container's rtslib-fb version with open-iscsi/rtslib-fb pull request 156 and reproduce the failure using the listed EL 8.8, Ceph 17.2.6, and Docker details. Done means the container includes the relevant fix and no longer enters the db_root error and restart loop.

Written by the indexing model from the issue text.