Unable to connect to NVMe-OF target

I followed the example in the README: expose a single Ceph rbd device, Ceph0, as a NVMe-OF target. For sake of completeness see the ouput of the gateway below:

```
[2022-10-06 08:35:05.831283] reactor.c: 943:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-06 08:35:05.831459] accel_engine.c: 510:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.
DEBUG:control.server:create_transport: tcp options:
[2022-10-06 08:35:06.154587] tcp.c: 617:nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
INFO:control.state:nvmeof.yolo.state omap object already exists.
DEBUG:control.state:Read key: omap_version -> 6
INFO:control.grpc:Received request to create bdev Ceph0 from rbd/mytestdevimage with block size 4096
[2022-10-06 08:35:06.339104] bdev_rbd.c:1253:bdev_rbd_create: *NOTICE*: Add Ceph0 rbd disk to lun
INFO:control.grpc:create_bdev: Ceph0
INFO:control.grpc:Received request to create subsystem nqn.2016-06.io.spdk:cnode1
INFO:control.grpc:create_subsystem nqn.2016-06.io.spdk:cnode1: True
INFO:control.grpc:Received request to add Ceph0 to nqn.2016-06.io.spdk:cnode1
INFO:control.grpc:add_namespace: 1
INFO:control.grpc:Received request to add host nqn.2014-08.org.nvmexpress:uuid:07ac0178-d457-415d-8ba8-bef911e64b88 to nqn.2016-06.io.spdk:cnode1
INFO:control.grpc:add_host nqn.2014-08.org.nvmexpress:uuid:07ac0178-d457-415d-8ba8-bef911e64b88: True
INFO:control.grpc:Received request to create TCP listener for nqn.2016-06.io.spdk:cnode1 at :5001.
[2022-10-06 08:35:06.437713] tcp.c: 819:nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on [2001:7b8:3000:999::11] port 5001 ***
INFO:control.grpc:create_listener: True
INFO:control.state:Restore complete.
INFO:control.server:Starting server...
```

Ask controller for state:
```
INFO:control.grpc:Received request to get subsystems
INFO:control.grpc:get_subsystems: [{'nqn': 'nqn.2014-08.org.nvmexpress.discovery', 'subtype': 'Discovery', 'listen_addresses': [], 'allow_any_host': True, 'hosts': []}, {'nqn': 'nqn.2016-06.io.spdk:cnode1', 'subtype': 'NVMe', 'listen_addresses': [{'transport': 'TCP', 'trtype': 'TCP', 'adrfam': 'IPv6', 'traddr': '[2001:7b8:3000:999::11]', 'trsvcid': '5001'}], 'allow_any_host': False, 'hosts': [{'nqn': 'nqn.2014-08.org.nvmexpress:uuid:07ac0178-d457-415d-8ba8-bef911e64b88'}], 'serial_number': 'SPDK00000000000001', 'model_number': 'SPDK bdev Controller', 'max_namespaces': 32, 'min_cntlid': 1, 'max_cntlid': 65519, 'namespaces': [{'nsid': 1, 'bdev_name': 'Ceph0', 'name': 'Ceph0', 'nguid': 'D65F908F8C7043EE966446B517F0F67C', 'uuid': 'd65f908f-8c70-43ee-9664-46b517f0f67c'}]}]
```
On client server discover the NVMe-OF target:

```nvme discover -t tcp -a 2001:7b8:3000:999::11 -s 5001

Discovery Log Number of Records 1, Generation counter 1
=====Discovery Log Entry 0======
trtype: tcp
adrfam: ipv6
subtype: nvme subsystem
treq: not required
portid: 0
trsvcid: 5001
subnqn: nqn.2016-06.io.spdk:cnode1
traddr: [2001:7b8:3000:999::11]
sectype: none
```
Make sure hostnqn matches access restriction:
```
cat /etc/nvme/hostnqn
nqn.2014-08.org.nvmexpress:uuid:07ac0178-d457-415d-8ba8-bef911e64b88
```

And try to connect to it:
```
~/ceph-nvmeof# nvme connect -t tcp --traddr 2001:7b8:3000:999::11 s 5001 -n nqn.2016-06.io.spdk:cnode1
Failed to write to /dev/nvme-fabrics: Connection refused
```
This fails:
```
~/ceph-nvmeof# nvme connect -t tcp --traddr 2001:7b8:3000:999::11 s 5001 -n nqn.2016-06.io.spdk:cnode1
Failed to write to /dev/nvme-fabrics: Connection refused

Kernel logs:

[74570.821638] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr [2001:07b8:3000:0999:0000:0000:0000:0011]:5001
[74570.822184] nvme nvme0: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[74708.070971] nvme nvme0: failed to connect socket: -111
```
I have tested kernels `5.4.0-126-generic` and `5.15.0-48-generic`.

Let's try if SPDK identify works:
```
~/ceph-nvmeof/spdk/build/examples# ./identify -r 'trtype:TCP adrfam:IPv6 traddr:2001:7b8:3000:999::11 trsvcid:5001'
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-06 08:49:56.655845] nvme_fabric.c: 180:nvme_fabric_prop_get_cmd_sync: *ERROR*: Property Get failed
[2022-10-06 08:49:56.687835] nvme_fabric.c: 180:nvme_fabric_prop_get_cmd_sync: *ERROR*: Property Get failed
=====================================================
NVMe over Fabrics controller at 2001:7b8:3000:999::11:5001: nqn.2014-08.org.nvmexpress.discovery
=====================================================
Controller Capabilities/Features
================================
Vendor ID: 0000
Subsystem Vendor ID: 0000
Serial Number: ....................
Model Number: ........................................
Firmware Version: 22.01
Recommended Arb Burst: 0
IEEE OUI Identifier: 00 00 00
Multi-path I/O
May have multiple subsystem ports: No
May have multiple controllers: No
Associated with SR-IOV VF: No
Max Data Transfer Size: 131072
Max Number of Namespaces: 0
NVMe Specification Version (VS): 1.3
NVMe Specification Version (Identify): 1.3
Maximum Queue Entries: 128
Contiguous Queues Required: Yes
Arbitration Mechanisms Supported
Weighted Round Robin: Not Supported
Vendor Specific: Not Supported
Reset Timeout: 15000 ms
Doorbell Stride: 4 bytes
NVM Subsystem Reset: Not Supported
Command Sets Supported
NVM Command Set: Supported
Boot Partition: Not Supported
Memory Page Size Minimum: 4096 bytes
Memory Page Size Maximum: 4096 bytes
Persistent Memory Region: Not Supported
Optional Asynchronous Events Supported
Namespace Attribute Notices: Not Supported
Firmware Activation Notices: Not Supported
128-bit Host Identifier: Not Supported

Controller Memory Buffer Support
================================
Supported: No

Persistent Memory Region Support
================================
Supported: No

Admin Command Set Attributes
============================
Security Send/Receive: Not Supported
Format NVM: Not Supported
Firmware Activate/Download: Not Supported
Namespace Management: Not Supported
Device Self-Test: Not Supported
Directives: Not Supported
NVMe-MI: Not Supported
Virtualization Management: Not Supported
Doorbell Buffer Config: Not Supported
Abort Command Limit: 1
Async Event Request Limit: 4
Number of Firmware Slots: N/A
Firmware Slot 1 Read-Only: N/A
Firmware Update Granularity: No Information Provided
Per-Namespace SMART Log: No
Asymmetric Namespace Access Log Page: Not Supported
Command Effects Log Page: Not Supported
Get Log Page Extended Data: Supported
Telemetry Log Pages: Not Supported
Error Log Page Entries Supported: 128
Keep Alive: Not Supported

NVM Command Set Attributes
==========================
Submission Queue Entry Size
Max: 1
Min: 1
Completion Queue Entry Size
Max: 1
Min: 1
Number of Namespaces: 0
Compare Command: Not Supported
Write Uncorrectable Command: Not Supported
Dataset Management Command: Not Supported
Write Zeroes Command: Not Supported
Set Features Save Field: Not Supported
Reservations: Not Supported
Timestamp: Not Supported
Copy: Not Supported
Volatile Write Cache: Not Present
Atomic Write Unit (Normal): 1
Atomic Write Unit (PFail): 1
Atomic Compare & Write Unit: 2
Fused Compare & Write: Supported
Scatter-Gather List
SGL Command Set: Supported
SGL Keyed: Supported
SGL Bit Bucket Descriptor: Not Supported
SGL Metadata Pointer: Not Supported
Oversized SGL: Not Supported
SGL Metadata Address: Not Supported
SGL Offset: Supported
Transport SGL Data Block: Not Supported
Replay Protected Memory Block: Not Supported

Firmware Slot Information
=========================
Active slot: 0

Error Log
=========

Active Namespaces
=================
Discovery Log Page
==================
Generation Counter: 1
Number of Records: 0
Record Format: 0
```
What stands out for me from above report is `Max Number of Namespaces: 0`. Can it be that for some reason the namespace is not exposed?

Let's ask SPDK through `rpc.py`:
```
~/ceph-nvmeof/spdk/scripts# ./rpc.py nvmf_get_subsystems
[
{
"nqn": "nqn.2014-08.org.nvmexpress.discovery",
"subtype": "Discovery",
"listen_addresses": [],
"allow_any_host": true,
"hosts": []
},
{
"nqn": "nqn.2016-06.io.spdk:cnode1",
"subtype": "NVMe",
"listen_addresses": [
{
"transport": "TCP",
"trtype": "TCP",
"adrfam": "IPv6",
"traddr": "[2001:7b8:3000:999::11]",
"trsvcid": "5001"
}
],
"allow_any_host": false,
"hosts": [
{
"nqn": "nqn.2014-08.org.nvmexpress:uuid:07ac0178-d457-415d-8ba8-bef911e64b88"
}
],
"serial_number": "SPDK00000000000001",
"model_number": "SPDK bdev Controller",
"max_namespaces": 32,
"min_cntlid": 1,
"max_cntlid": 65519,
"namespaces": [
{
"nsid": 1,
"bdev_name": "Ceph0",
"name": "Ceph0",
"nguid": "D65F908F8C7043EE966446B517F0F67C",
"uuid": "d65f908f-8c70-43ee-9664-46b517f0f67c"
}
]
}
]
```
There definitely is a namespace exposed. I'm at a loss at this point. Is there something obvious I'm overlooking? Note that I have also tried "127.0.0.1" as listener address and tried to connect with the kernel client on the host running the gateway, but this results in the same error.

Start with the README setup and reproduce the failure using nvme discover, nvme connect, and SPDK's identify entry point. Compare the controller state from rpc.py nvmf_get_subsystems with the kernel connection logs; done means the client connects to nqn.2016-06.io.spdk:cnode1 and exposes the Ceph0 namespace.

Written by the indexing model from the issue text.