[1.0.0] No /dev/nvme0n1 and null nvme list on EL9.3 client with --enable-ha and --ana-reporting

Hello,

In order to evaluate how ceph-nvmeof behaves in a HA setup, I've set up 2 GWs as per below:

ceph-nvmeof.conf of GW #1 test-lis04h02:

```
# This file is generated by cephadm.
[gateway]
name = client.nvmeof.nvmeof_pool01.test-lis04h02.baakhx
group = None
addr = 100.74.191.134
port = 5500
enable_auth = False
state_update_notify = True
state_update_interval_sec = 5
enable_spdk_discovery_controller = true
enable_prometheus_exporter = false

[discovery]
addr = 100.74.191.134
port = 8009

[ceph]
pool = nvmeof_pool01
config_file = /etc/ceph/ceph.conf
id = nvmeof.nvmeof_pool01.test-lis04h02.baakhx

#[mtls]
#server_key = ./server.key
#client_key = ./client.key
#server_cert = ./server.crt
#client_cert = ./client.crt

[spdk]
tgt_path = /usr/local/bin/nvmf_tgt
rpc_socket = /var/tmp/spdk.sock
timeout = 60
log_level = WARN
conn_retries = 10
transports = tcp
transport_tcp_options = {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}
tgt_cmd_extra_args = --cpumask=0xF

```

ceph-nvmeof.conf of GW #2 test-mom02h02:

```
# This file is generated by cephadm.
[gateway]
name = client.nvmeof.nvmeof_pool01.test-mom02h02.lcdzzc
group = None
addr = 100.74.191.130
port = 5500
enable_auth = False
state_update_notify = True
state_update_interval_sec = 5
enable_spdk_discovery_controller = true
enable_prometheus_exporter = false

[discovery]
addr = 100.74.191.130
port = 8009

[ceph]
pool = nvmeof_pool01
config_file = /etc/ceph/ceph.conf
id = nvmeof.nvmeof_pool01.test-mom02h02.lcdzzc

#[mtls]
#server_key = ./server.key
#client_key = ./client.key
#server_cert = ./server.crt
#client_cert = ./client.crt

[spdk]
tgt_path = /usr/local/bin/nvmf_tgt
rpc_socket = /var/tmp/spdk.sock
timeout = 60
log_level = WARN
conn_retries = 10
transports = tcp
transport_tcp_options = {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}
tgt_cmd_extra_args = --cpumask=0xF
```

Deleting rados config object:

```
rados -p nvmeof_pool01 rm nvmeof.None.state
```

Started two [1.0.0] containers on nodes test-lis04h02 and test-mom02h02 with below options:

```
/usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --init --name ceph-aa558815-042c-4fce-ac37-80c0255bf3c0-nvmeof-nvmeof_pool01-test-lis04h02-baakhx --pids-limit=-1 --ulimit memlock=-1:-1 --ulimit nofile=10240 --cap-add=SYS_ADMIN --cap-add=CAP_SYS_NICE --log-driver journald --conmon-pidfile /run/ceph-aa558815-042c-4fce-ac37-80c0255bf3c0@nvmeof.nvmeof_pool01.test-lis04h02.baakhx.service-pid --cidfile /run/ceph-aa558815-042c-4fce-ac37-80c0255bf3c0@nvmeof.nvmeof_pool01.test-lis04h02.baakhx.service-cid --cgroups=split -e CONTAINER_IMAGE=quay.io/ceph/nvmeof:1.0.0 -e NODE_NAME=test-lis04h02.peta.libe.dc.univ-lorraine.fr -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-lis04h02.baakhx/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-lis04h02.baakhx/keyring:/etc/ceph/keyring:z -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-lis04h02.baakhx/ceph-nvmeof.conf:/src/ceph-nvmeof.conf:z -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-lis04h02.baakhx/configfs:/sys/kernel/config -v /dev/hugepages:/dev/hugepages -v /dev/vfio/vfio:/dev/vfio/vfio -v /etc/hosts:/etc/hosts:ro --mount type=bind,source=/lib/modules,destination=/lib/modules,ro=true quay.io/ceph/nvmeof:1.0.0
```

```
/usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --init --name ceph-aa558815-042c-4fce-ac37-80c0255bf3c0-nvmeof-nvmeof_pool01-test-mom02h02-lcdzzc --pids-limit=-1 --ulimit memlock=-1:-1 --ulimit nofile=10240 --cap-add=SYS_ADMIN --cap-add=CAP_SYS_NICE --log-driver journald --conmon-pidfile /run/ceph-aa558815-042c-4fce-ac37-80c0255bf3c0@nvmeof.nvmeof_pool01.test-mom02h02.lcdzzc.service-pid --cidfile /run/ceph-aa558815-042c-4fce-ac37-80c0255bf3c0@nvmeof.nvmeof_pool01.test-mom02h02.lcdzzc.service-cid --cgroups=split -e CONTAINER_IMAGE=quay.io/ceph/nvmeof:1.0.0 -e NODE_NAME=test-mom02h02.peta.montet.dc.univ-lorraine.fr -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-mom02h02.lcdzzc/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-mom02h02.lcdzzc/keyring:/etc/ceph/keyring:z -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-mom02h02.lcdzzc/ceph-nvmeof.conf:/src/ceph-nvmeof.conf:z -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-mom02h02.lcdzzc/configfs:/sys/kernel/config -v /dev/hugepages:/dev/hugepages -v /dev/vfio/vfio:/dev/vfio/vfio -v /etc/hosts:/etc/hosts:ro --mount type=bind,source=/lib/modules,destination=/lib/modules,ro=true quay.io/ceph/nvmeof:1.0.0
```

And configured the GWs:

```
podman run -it quay.io/ceph/nvmeof-cli:1.0.0 --server-address 100.74.191.134 --server-port 5500 subsystem add --subsystem "nqn.2024-02.io.spdk:peta" --enable-ha --ana-reporting
Adding subsystem nqn.2024-02.io.spdk:peta: Successful

podman run -it quay.io/ceph/nvmeof-cli:1.0.0 --server-address 100.74.191.134 --server-port 5500 namespace add --subsystem "nqn.2024-02.io.spdk:peta" --rbd-pool nvmeof_pool01 --rbd-image image-1 --size 3G --rbd-create-image --nsid 1 --load-balancing-group 1
Adding namespace 1 to nqn.2024-02.io.spdk:peta, load balancing group 1: Successful

podman run -it quay.io/ceph/nvmeof-cli:1.0.0 --server-address 100.74.191.134 --server-port 5500 listener add --subsystem "nqn.2024-02.io.spdk:peta" --gateway-name client.nvmeof.nvmeof_pool01.test-lis04h02.baakhx --traddr 100.74.191.134 --trsvcid 4420
Adding nqn.2024-02.io.spdk:peta listener at 100.74.191.134:4420: Successful

podman run -it quay.io/ceph/nvmeof-cli:1.0.0 --server-address 100.74.191.130 --server-port 5500 listener add --subsystem "nqn.2024-02.io.spdk:peta" --gateway-name client.nvmeof.nvmeof_pool01.test-mom02h02.lcdzzc --traddr 100.74.191.130 --trsvcid 4420
Adding nqn.2024-02.io.spdk:peta listener at 100.74.191.130:4420: Successful

podman run -it quay.io/ceph/nvmeof-cli:1.0.0 --server-address 100.74.191.130 --server-port 5500 host add --subsystem "nqn.2024-02.io.spdk:peta" --host "*"
Allowing open host access to nqn.2024-02.io.spdk:peta: Successful
```

suybsystem and namespace configuration check:

```
$ podman run -it quay.io/ceph/nvmeof-cli:1.0.0 --server-address 100.74.191.130 --server-port 5500 subsystem list
Subsystems:
╒═══════════╤══════════════════════════════════════╤════════════╤════════════════════╤══════════════════════╤══════════════════╤═════════════╕
│ Subtype │ NQN │ HA State │ Serial Number │ Model Number │ Controller IDs │ Namespace │
│ │ │ │ │ │ │ Count │
╞═══════════╪══════════════════════════════════════╪════════════╪════════════════════╪══════════════════════╪══════════════════╪═════════════╡
│ Discovery │ nqn.2014-08.org.nvmexpress.discovery │ disabled │ │ │ 0-0 │ 0 │
├───────────┼──────────────────────────────────────┼────────────┼────────────────────┼──────────────────────┼──────────────────┼─────────────┤
│ NVMe │ nqn.2024-02.io.spdk:peta │ enabled │ SPDK77352934285842 │ SPDK bdev Controller │ 1-65519 │ 1 │
╘═══════════╧══════════════════════════════════════╧════════════╧════════════════════╧══════════════════════╧══════════════════╧═════════════╛

$ podman run -it quay.io/ceph/nvmeof-cli:1.0.0 --server-address 100.74.191.134 --server-port 5500 subsystem list
Subsystems:
╒═══════════╤══════════════════════════════════════╤════════════╤════════════════════╤══════════════════════╤══════════════════╤═════════════╕
│ Subtype │ NQN │ HA State │ Serial Number │ Model Number │ Controller IDs │ Namespace │
│ │ │ │ │ │ │ Count │
╞═══════════╪══════════════════════════════════════╪════════════╪════════════════════╪══════════════════════╪══════════════════╪═════════════╡
│ Discovery │ nqn.2014-08.org.nvmexpress.discovery │ disabled │ │ │ 0-0 │ 0 │
├───────────┼──────────────────────────────────────┼────────────┼────────────────────┼──────────────────────┼──────────────────┼─────────────┤
│ NVMe │ nqn.2024-02.io.spdk:peta │ enabled │ SPDK77352934285842 │ SPDK bdev Controller │ 1-65519 │ 1 │
╘═══════════╧══════════════════════════════════════╧════════════╧════════════════════╧══════════════════════╧══════════════════╧═════════════╛

$ podman run -it quay.io/ceph/nvmeof-cli:1.0.0 --server-address 100.74.191.130 --server-port 5500 namespace list -n "nqn.2024-02.io.spdk:peta"
Namespaces in subsystem nqn.2024-02.io.spdk:peta:
╒════════╤════════════════════════╤═══════════════╤═════════╤═════════╤═════════╤═════════════════════╤═════════════╤═══════════╤═══════════╤════════════╤═════════════╕
│ NSID │ Bdev │ RBD │ RBD │ Image │ Block │ UUID │ Load │ R/W IOs │ R/W MBs │ Read MBs │ Write MBs │
│ │ Name │ Pool │ Image │ Size │ Size │ │ Balancing │ per │ per │ per │ per │
│ │ │ │ │ │ │ │ Group │ second │ second │ second │ second │
╞════════╪════════════════════════╪═══════════════╪═════════╪═════════╪═════════╪═════════════════════╪═════════════╪═══════════╪═══════════╪════════════╪═════════════╡
│ 1 │ bdev_96ae08eb-fa83- │ nvmeof_pool01 │ image-1 │ 3 GiB │ 512 B │ 96ae08eb-fa83-4fd8- │ 1 │ unlimited │ unlimited │ unlimited │ unlimited │
│ │ 4fd8-8f82-2e9fee187e91 │ │ │ │ │ 8f82-2e9fee187e91 │ │ │ │ │ │
╘════════╧════════════════════════╧═══════════════╧═════════╧═════════╧═════════╧═════════════════════╧═════════════╧═══════════╧═══════════╧════════════╧═════════════╛
```

What I observe: /dev/nvme0n1 is not created on test-mom02h01 EL 9.3 client.

```
$ nvme discover -t tcp -a 100.74.191.134 -s 4420

Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype: tcp
adrfam: ipv4
subtype: nvme subsystem
treq: not required
portid: 0
trsvcid: 4420
subnqn: nqn.2024-02.io.spdk:peta
traddr: 100.74.191.134
eflags: none
sectype: none

$ nvme connect -t tcp -a 100.74.191.134 -n nqn.2024-02.io.spdk:peta

$ nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------

$ nvme list -v
Subsystem Subsystem-NQN Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys0 nqn.2024-02.io.spdk:peta nvme0

Device SN MN FR TxPort Address Subsystem Namespaces
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
nvme0 SPDK77352934285842 SPDK bdev Controller 23.01.1 tcp traddr=100.74.191.134,trsvcid=4420,src_addr=100.74.191.129 nvme-subsys0

Device Generic NSID Usage Format Controllers
------------ ------------ ---------- -------------------------- ---------------- ----------------

$ ls -alh /dev/nvme*
crw------- 1 root root 241, 0 23 févr. 11:03 /dev/nvme0
crw------- 1 root root 10, 123 23 févr. 10:21 /dev/nvme-fabrics

$ fdisk -l /dev/nvme0
fdisk: impossible d'ouvrir /dev/nvme0: Repérage non permis

$ dmesg
[...]
[61797.892756] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 100.74.191.134:4420
[61797.895072] nvme nvme0: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[61802.258173] nvme nvme0: creating 4 I/O queues.
[61802.297181] nvme nvme0: mapped 4/0/0 default/read/poll queues.
[61802.299326] nvme nvme0: new ctrl: NQN "nqn.2024-02.io.spdk:peta", addr 100.74.191.134:4420
```

/dev/nvme0n1 is not created on the client.

What I expect: /dev/nvme0n1 being created and usable on the client.

Note: reproducing these exact commands without using --enable-ha and --ana-reporting options when creating the subsystem results in /dev/nvme0n1 being created on the client as expected.

Start by reproducing the EL9.3 client behavior with the two ceph-nvmeof 1.0.0 gateways using the shown nvme discover, connect, list, and list -v commands, and review the referenced ceph-nvmeof.conf settings. Done means the HA/ANA connection exposes /dev/nvme0n1 and nvme list reports the namespace instead of an empty or null device list.

Written by the indexing model from the issue text.