[Bug]: NIXL cache-transceiver hangs forever if a single completion notification is dropped

System Info

• TRT-LLM version: v1.3.0rc0, confirmed unchanged in v1.3.0rc24 and main
• GPUs: GB200 / GB300
• cache_transceiver_config.backend: NIXL, disaggregated serving (separate CTX/GEN), dep32 attention-DP configs

Reproduction

AgentConnection::send() transfers KV cache data and signals completion in two separate NIXL calls:

// cpp/tensorrt_llm/executor/cache_transmission/agent_utils/connection.cpp:127-148
auto status = mAgentConnectionManager->getAgent()->submitTransferRequests(request);
TransferState transferState = status->wait();
TLLM_CHECK_WITH_INFO(transferState == TransferState::kSUCCESS, "AgentConnection::send failed");
// TODO: there is a bug in request_with_notify https://github.com/ai-dynamo/nixl/pull/252
mAgentConnectionManager->getAgent()->notifySyncMessage(mRemoteAgentName, ss.str());

The data goes over RDMA, the "done" signal goes over NIXL's notification channel as an independent message. NIXL PR #252 (linked in the TODO above) fixed the case where this notification arrives before the data does. It does not guarantee the notification is never dropped.

The receive side waits for that notification with no timeout and no retry:

// cpp/tensorrt_llm/executor/cache_transmission/agent_utils/connection.cpp:495-539
bool AgentConnectionManager::waitForNotification(...)
{
    while (!terminateFlag.load())
    {
        updateUnhandledNotifications();  // polls getNotifiedSyncMessages(), no sleep
        ...
    }
    return false;
}

terminateFlag only flips on executor shutdown, not on a stuck transfer. If the notification packet is dropped, this spins forever at 100% CPU while the GPU sits idle. Same code path is shared by the Mooncake backend (AgentConnectionManager).

We ruled out hardware first (per-rail ib_write_bw, ucx_perftest -m cuda, UCX transport route logs, dmesg — all clean) before tracing it to this code.

We first hit this 100% reproducing on one rack, worked around it by switching that rack to the UCX backend. Weeks later the same config family hit the identical hang on a different rack, still on NIXL. Looks like a timing-dependent race in notification delivery, not a rack-specific hardware issue — it just needs the right fabric conditions to trigger.

Actual behavior

GEN worker goes silent after model init, no error, no crash:

3:     self.kv_cache_transceiver.check_gen_transfer_status(atLeastNum)
3:   File ".../kv_cache_transceiver.py", line 155, in check_gen_transfer_status
3:     signaled = self._cond.wait(timeout)

CTX side eventually hits its own per-request kv_transfer_timeout_ms and terminates every in-flight request:

0: [RANK 0] [W] Terminating context request 34 due to KV cache transfer timeout

Nothing on the GEN side ever times out or errors. The job runs until an external watchdog kills it.

Expected behavior

A dropped notification should eventually fail the request, not hang forever.

I see cache_transceiver_config.kv_transfer_timeout_ms + TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1 already exist in main and do wire a real per-request cancel flag down through this exact wait loop via consensus across ranks (CacheReceiver::cancelRequest / CacheSender::cancelRequest). That's presumably the intended fix for this class of hang. Two gaps:

1. It's undiscoverable from this code path — nothing in waitForNotification() or the two AgentConnection methods above points at it.
2. It's opt-in. Without the flag, kv_transfer_timeout_ms alone is observe-only (logs a warning, cancels nothing), and waitForNotification() is exactly as unbounded as shown above.

Worth at minimum a comment on waitForNotification() and AgentConnection::send/recv pointing at TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL, and worth asking why this protection isn't the default rather than opt-in/experimental.

Start in cpp/tensorrt_llm/executor/cache_transmission/agent_utils/connection.cpp, reading AgentConnection::send and recv alongside AgentConnectionManager::waitForNotification. Trace how kv_transfer_timeout_ms and TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL reach the wait loop, then verify that a dropped notification no longer leaves the request unbounded and that the relevant protection is discoverable or appropriately enabled.