[Bug] TRTLLM_NIXL_KVCACHE_BACKEND causes rail endpoint serialization mismatch with LIBFABRIC backend

Description

When using TensorRT-LLM 1.2.0rc5 with NIXL for KV cache transfer over AWS EFA using the LIBFABRIC backend, setting both NIXL_BACKEND and TRTLLM_NIXL_KVCACHE_BACKEND environment variables causes a rail endpoint serialization mismatch that prevents successful KV cache transfers between prefill and decode workers.

Environment

• TensorRT-LLM Version: 1.2.0rc5
• NIXL Version: Bundled with TRT-LLM 1.2.0rc5
• Platform: AWS p5.48xlarge (8x H100 GPUs, 32 EFA devices)
• OS: Ubuntu 22.04 (NVIDIA container)
• CUDA: 12.8 / 13.0
• Use Case: Disaggregated inference with prefill/decode workers using KV cache transfer over EFA

Error Symptoms

When TRTLLM_NIXL_KVCACHE_BACKEND=LIBFABRIC is set alongside NIXL_BACKEND=LIBFABRIC:

Deserialization of tag dest_data_ep_4 failed
Key dest_data_ep_4 not found or has invalid length: -1
Failed to deserialize data rail endpoints
NIXL_ERR_BACKEND

The KV cache transfer fails, and inference requests hang or fail.

Root Cause Analysis

TensorRT-LLM with NIXL has a two-layer backend configuration system:

Layer 1: TRT-LLM C++ Layer

• Environment Variable: TRTLLM_NIXL_KVCACHE_BACKEND
• Source: tensorrt_llm/executor/kvCacheTransferAgent.cpp
• Function: getEnvNixlBackend()
• Log Message: NixlTransferAgent using NIXL backend: <BACKEND>

Layer 2: Dynamo Python Layer

• Environment Variable: NIXL_BACKEND
• Source: dynamo/nixl_connect/__init__.py
• Class: Connector
• Log Message: NIXL Connect: Using backend(s): ['<BACKEND>']

The Mismatch Problem

When TRTLLM_NIXL_KVCACHE_BACKEND=LIBFABRIC is set:

1. TRT-LLM C++ layer (kvCacheTransferAgent.cpp) creates 32 rail endpoints for the 32 EFA devices on p5.48xlarge
2. Each endpoint is serialized with names like dest_data_ep_0, dest_data_ep_1, ... dest_data_ep_31
3. The Dynamo Python layer expects a different endpoint naming scheme
4. Deserialization fails because the expected keys (dest_data_ep_4) don't match what was serialized

Why Single-Layer Works

When only NIXL_BACKEND=LIBFABRIC is set (without TRTLLM_NIXL_KVCACHE_BACKEND):

1. The TRT-LLM C++ layer uses its default backend (UCX)
2. The Dynamo Python layer handles LIBFABRIC backend selection
3. Endpoint naming is consistent because only one layer is configuring the backend
4. KV cache transfers succeed

Steps to Reproduce

Broken Configuration

env:
  - name: NIXL_BACKEND
    value: "LIBFABRIC"
  - name: TRTLLM_NIXL_KVCACHE_BACKEND
    value: "LIBFABRIC"  # <-- CAUSES MISMATCH
  - name: FI_PROVIDER
    value: "efa"
  - name: FI_EFA_USE_DEVICE_RDMA
    value: "1"
  - name: FI_HMEM_DISABLE_P2P
    value: "0"

Working Configuration

env:
  - name: NIXL_BACKEND
    value: "LIBFABRIC"
  # DO NOT SET: TRTLLM_NIXL_KVCACHE_BACKEND
  - name: FI_PROVIDER
    value: "efa"
  - name: FI_EFA_USE_DEVICE_RDMA
    value: "1"
  - name: FI_HMEM_DISABLE_P2P
    value: "1"

Expected Behavior

Setting both NIXL_BACKEND=LIBFABRIC and TRTLLM_NIXL_KVCACHE_BACKEND=LIBFABRIC should either:

1. Work correctly with consistent rail endpoint serialization, OR
2. Warn users that only one should be set, OR
3. Have the C++ layer defer to the Python layer's backend configuration when using LIBFABRIC

Actual Behavior

Setting both environment variables causes the C++ layer and Python layer to create incompatible rail endpoint serialization, resulting in NIXL_ERR_BACKEND errors.

Workaround

Only set NIXL_BACKEND=LIBFABRIC. Do NOT set TRTLLM_NIXL_KVCACHE_BACKEND when using LIBFABRIC backend.

Deployment History (Testing Evidence)

Version	Config	TRTLLM_NIXL_KVCACHE_BACKEND	FI_HMEM_DISABLE_P2P	Result
v37	Single-layer	NOT SET	1	✅ WORKING
v38	Two-layer	LIBFABRIC	1	❌ FAILED
v39	UCX only	NOT SET	N/A	✅ WORKING
v40	Two-layer + P2P fix	LIBFABRIC	0	❌ FAILED
v41	Single-layer (v37 config)	NOT SET	1	✅ WORKING

Additional Context

• This bug only affects LIBFABRIC backend. UCX backend works correctly with both environment variables set.
• The issue is specifically in the rail endpoint serialization/deserialization between the two layers.
• On p5.48xlarge, there are 32 EFA devices which creates 32 rail endpoints, making this more likely to hit serialization issues.

Related

• AWS EFA with LIBFABRIC provider
• NIXL KV cache transceiver
• Disaggregated inference (prefill/decode separation)

Read getEnvNixlBackend() in tensorrt_llm/executor/kvCacheTransferAgent.cpp and the Connector class in dynamo/nixl_connect/init.py, then compare how both layers serialize and deserialize LIBFABRIC rail endpoints. Reproduce the broken and working environment configurations on the documented AWS setup. Done means the two-layer configuration no longer produces NIXL_ERR_BACKEND, or clearly warns that only one backend variable should be set.