ceph / ceph/ceph-nvmeof

Failed to create large iobuf pool

Open
#239 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
132
Forks
69
Avg merge
1d 7h
Merged PRs (30d)
47

Description

nvmeof GW Deployment failed with below error messages.
```
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:Starting gateway client.nvmeof.rbd.ceph-1sunilkumar-qimeq6-node5.nintap
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: DEBUG:control.server:Starting serve
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: DEBUG:control.server:Configuring server client.nvmeof.rbd.ceph-1sunilkumar-qimeq6-node5.nintap
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:SPDK Target Path: /usr/local/bin/nvmf_tgt
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:SPDK Socket: /var/tmp/spdk.sock
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:Starting /usr/local/bin/nvmf_tgt -u -r /var/tmp/spdk.sock
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:control.server:Attempting to initialize SPDK: rpc_socket: /var/tmp/spdk.sock, conn_retries: 300, timeout: 60.0
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO: Setting log level to WARN
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: INFO:JSONRPCClient(/var/tmp/spdk.sock):Setting log level to WARN
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.436592] Starting SPDK v23.01.1 / DPDK 22.11.1 initialization...
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.436787] [ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --no-pci --huge-unlink --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid3 ]
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: TELEMETRY: No legacy callbacks, legacy socket not created
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.550555] app.c: 712:spdk_app_start: *NOTICE*: Total cores available: 1
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.602641] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.652634] thread.c:2956:spdk_iobuf_initialize: *ERROR*: Failed to create large iobuf pool
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670091] iobuf.c: 54:iobuf_subsystem_initialize: *ERROR*: Failed to initialize iobuf
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670188] subsystem.c: 143:spdk_subsystem_init_next: *ERROR*: Init subsystem iobuf failed
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670201] app.c: 819:spdk_app_stop: *WARNING*: spdk_app_stop'd on non-zero
Sep 21 09:03:37 ceph-1sunilkumar-qimeq6-node5 ceph-b52edbca-5869-11ee-b40d-fa163e7c4502-nvmeof-rbd-ceph-1sunilkumar-qimeq6-node5-nintap[61564]: [2023-09-21 13:03:37.670221] thread.c:2184:spdk_io_device_unregister: *ERROR*: io_device 0x708b20 not found
```
[journal1.log](https://github.com/ceph/ceph-nvmeof/files/12685414/journal1.log)

Contributor guide

Open the contributing guide

Research direction

Start with journal1.log and the reported SPDK initialization sequence, focusing on the failure at spdk_iobuf_initialize and the preceding DPDK parameters. Reproduce the NVMe-oF gateway deployment in a comparable environment and determine the cause of the large iobuf pool failure. Done means the gateway starts successfully without the iobuf initialization errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.