Failed to create large iobuf pool
- Dominant language
- Python
- Stars
- 132
- Forks
- 69
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 47
Description
nvmeof GW Deployment failed with below error messages.
```
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:Starting gateway client.nvmeof.rbd.ceph-1sunilkumar-qimeq6-node5.nintap
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: DEBUG:control.server:Starting serve
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: DEBUG:control.server:Configuring server client.nvmeof.rbd.ceph-1sunilkumar-qimeq6-node5.nintap
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:SPDK Target Path: /usr/local/bin/nvmf_tgt
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:SPDK Socket: /var/tmp/spdk.sock
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:Starting /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:Attempting to initialize SPDK: rpc_socket: /var/tmp/spdk.sock, conn_retries: 300, timeout: 60.0
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO: Setting log level to WARN
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:JSONRPCClient(/var/tmp/spdk.sock):Setting log level to WARN
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.436592] Starting SPDK v23.01.1 / DPDK 22.11.1 initialization...
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.436787] [ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --no-pci --huge-unlink --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid3 ]
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: TELEMETRY: No legacy callbacks, legacy socket not created
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.550555] app.c: 712:spdk_app_start: *NOTICE*: Total cores available: 1
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.602641] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.652634] thread.c:2956:spdk_iobuf_initialize: *ERROR*: Failed to create large iobuf pool
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670091] iobuf.c: 54:iobuf_subsystem_initialize: *ERROR*: Failed to initialize iobuf
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670188] subsystem.c: 143:spdk_subsystem_init_next: *ERROR*: Init subsystem iobuf failed
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670201] app.c: 819:spdk_app_stop: *WARNING*: spdk_app_stop'd on non-zero
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670221] thread.c:2184:spdk_io_device_unregister: *ERROR*: io_device 0x708b20 not found
```
[journal1.log](https://github.com/ceph/ceph-nvmeof/files/12685414/journal1.log)
Contributor guide
Research direction
Start with journal1.log and the reported SPDK initialization sequence, focusing on the failure at spdk_iobuf_initialize and the preceding DPDK parameters. Reproduce the NVMe-oF gateway deployment in a comparable environment and determine the cause of the large iobuf pool failure. Done means the gateway starts successfully without the iobuf initialization errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100