nixl._bindings.nixlBackendError: NIXL_ERR_BACKEND because of register_memory failure

### 🐛 Describe the bug

```
kubectl logs -f vllm-1p1d-roleset-sf8w4-decode-85b9745648-0 -p
INFO 08-10 22:40:52 [__init__.py:244] Automatically detected platform cuda.
WARNING 08-10 22:40:55 [api_server.py:913] SECURITY WARNING: Development endpoints are enabled! This should NOT be used in production!
INFO 08-10 22:40:56 [api_server.py:1395] vLLM API server version 0.9.2
INFO 08-10 22:40:56 [cli_args.py:325] non-default args: {'host': '0.0.0.0', 'uvicorn_log_level': 'warning', 'model': '/models/Qwen3-8B', 'served_model_name': ['qwen3-8B'], 'kv_transfer_config': KVTransferConfig(kv_connector='NixlConnector', engine_id='136233f3-e2f2-4717-9c6c-17271259102b', kv_buffer_device='cuda', kv_buffer_size=1000000000.0, kv_role='kv_both', kv_rank=None, kv_parallel_size=1, kv_ip='127.0.0.1', kv_port=14579, kv_connector_extra_config={}, kv_connector_module_path=None)}
WARNING 08-10 22:40:56 [config.py:463] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
INFO 08-10 22:41:01 [config.py:841] This model supports multiple tasks: {'classify', 'embed', 'generate', 'reward'}. Defaulting to 'generate'.
INFO 08-10 22:41:01 [config.py:1472] Using max model len 40960
INFO 08-10 22:41:01 [config.py:2285] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 08-10 22:41:05 [__init__.py:244] Automatically detected platform cuda.
WARNING 08-10 22:41:06 [api_server.py:913] SECURITY WARNING: Development endpoints are enabled! This should NOT be used in production!
INFO 08-10 22:41:06 [core.py:526] Waiting for init message from front-end.
INFO 08-10 22:41:06 [core.py:69] Initializing a V1 LLM engine (v0.9.2) with config: model='/models/Qwen3-8B', speculative_config=None, tokenizer='/models/Qwen3-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=qwen3-8B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
INFO 08-10 22:41:08 [parallel_state.py:1076] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 08-10 22:41:08 [nixl_connector.py:50] NIXL is available
INFO 08-10 22:41:08 [factory.py:74] Creating v1 connector with name: NixlConnector and engine_id: 136233f3-e2f2-4717-9c6c-17271259102b
INFO 08-10 22:41:08 [nixl_connector.py:345] Initializing NIXL wrapper
INFO 08-10 22:41:08 [nixl_connector.py:346] Initializing NIXL worker 136233f3-e2f2-4717-9c6c-17271259102b
Backend UCX was instantiated
Initialized NIXL agent: bd89bde0-1ea7-4c18-aab7-dfe92a562004
INFO 08-10 22:41:08 [cuda.py:284] Using Flash Attention backend on V1 engine.
INFO 08-10 22:41:08 [topk_topp_sampler.py:49] Using FlashInfer for top-p & top-k sampling.
INFO 08-10 22:41:08 [gpu_model_runner.py:1770] Starting to load model /models/Qwen3-8B...
INFO 08-10 22:41:08 [gpu_model_runner.py:1775] Loading model from scratch...
Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00refcount == 0' failed: region 0x15ca5770 0x7f2094000000..0x7f210d400000 of ucp_rcache
[1754890928.567126] [vllm-1p1d-roleset-sf8w4-decode-85b9745648-0:270 :0] rcache.c:697 UCX WARN ucp_rcache: destroying inuse region 0x15ca5770 [0x7f2094000000..0x7f210d400000] g- rw ref 1 md[5]=cuda_ipc md[6]=mlx5_7 md[7]=mlx5_7
==== backtrace (tid: 270) ====
0 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/../../nixl.libs/libucs-311e600f.so.0.0.0(ucs_handle_error+0x294) [0x7f28646daba4]
1 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/../../nixl.libs/libucs-311e600f.so.0.0.0(ucs_fatal_error_message+0xb2) [0x7f28646d7d22]
2 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/../../nixl.libs/libucs-311e600f.so.0.0.0(ucs_fatal_error_format+0xd7) [0x7f28646d7e07]
3 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/../../nixl.libs/libucs-311e600f.so.0.0.0(ucs_mem_region_destroy_internal+0x31e) [0x7f28646e1efe]
4 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/../../nixl.libs/libucs-311e600f.so.0.0.0(+0x40808) [0x7f28646e2808]
5 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/../../nixl.libs/libucs-311e600f.so.0.0.0(ucs_class_call_cleanup_chain+0x66) [0x7f28646f28a6]
6 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/../../nixl.libs/libucs-311e600f.so.0.0.0(ucs_rcache_destroy+0x18) [0x7f28646e5098]
7 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/../../nixl.libs/libucp-7a15df9c.so.0.0.0(ucp_mem_rcache_cleanup+0x18) [0x7f28586adaf8]
8 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/../../nixl.libs/libucp-7a15df9c.so.0.0.0(ucp_cleanup+0x16) [0x7f2858687436]
9 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/../libucx_utils.so(_ZN14nixlUcxContextD1Ev+0xc) [0x7f2864741aec]
10 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/libplugin_UCX.so(_ZN13nixlUcxEngineD2Ev+0x2cf) [0x7f28647a73bf]
11 /usr/local/lib/python3.12/dist-packages/.nixl.mesonpy.libs/plugins/libplugin_UCX.so(_ZN13nixlUcxEngineD0Ev+0x9) [0x7f28647a7589]
12 /usr/local/lib/python3.12/dist-packages/nixl/../.nixl.mesonpy.libs/libnixl.so(_ZN13nixlAgentDataD2Ev+0x97) [0x7f28902880f7]
13 /usr/local/lib/python3.12/dist-packages/nixl/../.nixl.mesonpy.libs/libnixl.so(_ZN9nixlAgentD1Ev+0xcb) [0x7f289028879b]
14 /usr/local/lib/python3.12/dist-packages/nixl/_bindings.cpython-312-x86_64-linux-gnu.so(+0x2db1e) [0x7f2ad009db1e]
15 /usr/local/lib/python3.12/dist-packages/nixl/_bindings.cpython-312-x86_64-linux-gnu.so(+0x48857) [0x7f2ad00b8857]
16 /usr/local/lib/python3.12/dist-packages/nixl/_bindings.cpython-312-x86_64-linux-gnu.so(+0x49201) [0x7f2ad00b9201]
17 /usr/bin/python3() [0x59b860]
18 /usr/bin/python3() [0x53bd64]
19 /usr/bin/python3() [0x59bded]
20 /usr/bin/python3() [0x59b8bb]
21 /usr/bin/python3() [0x53bfa9]
22 /usr/bin/python3() [0x63c24a]
23 /usr/bin/python3() [0x557c4e]
24 /usr/bin/python3() [0x6594c0]
25 /usr/bin/python3(Py_FinalizeEx+0x16c) [0x63f66c]
26 /usr/bin/python3(Py_Exit+0xc) [0x65960c]
27 /usr/bin/python3() [0x6570eb]
28 /usr/bin/python3() [0x656f14]
29 /usr/bin/python3(PyRun_SimpleStringFlags+0x56) [0x64ffb6]
30 /usr/bin/python3(Py_RunMain+0x4b2) [0x64d5a2]
31 /usr/bin/python3(Py_BytesMain+0x2d) [0x6064ad]
32 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f2d4c6f8d90]
33 /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f2d4c6f8e40]
34 /usr/bin/python3(_start+0x25) [0x606325]
=================================
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1495, in
uvloop.run(run_server(args))
File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1431, in run_server
await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1451, in run_server_worker
async with build_async_engine_client(args, client_config) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 158, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 194, in build_async_engine_client_from_engine_args
async_llm = AsyncLLM.from_vllm_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 162, in from_vllm_config
return cls(
^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 124, in __init__
self.engine_core = EngineCoreClient.make_async_mp_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 96, in make_async_mp_client
return AsyncMPClient(*client_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 666, in __init__
super().__init__(
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 403, in __init__
with launch_core_engines(vllm_config, executor_class,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
next(self.gen)
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 434, in launch_core_engines
wait_for_engine_startup(
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 484, in wait_for_engine_startup
raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

```

I used nodeselector in testing.

```
+ nodeSelector:
+ kubernetes.io/hostname: 192.168.0.6
```

### Steps to Reproduce

```
kubectl apply -f samples/disaggregation/vllm/1p1d.yaml
```

### Expected behavior

it should start to run

### Environment

nightly

Start with vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py at register_kv_caches and nixl/_api.py at register_memory, then review the UCX No such device log. Reproduce the Kubernetes failure and trace the NIXL memory-registration path; done means the engine initializes its KV cache without NIXL_ERR_BACKEND.

Written by the indexing model from the issue text.