NixlTransferAgent hardcodes /tmp/trtllm_nixl_port.lock — breaks shared-/tmp setups (Docker bind-mounts, parallel CI)

Summary

NixlTransferAgent hardcodes the path /tmp/trtllm_nixl_port.lock for its port-allocation file lock, with no env override. This makes it impossible to run two unrelated trtllm processes on the same host when they share a /tmp filesystem (Docker bind-mounts, parallel CI on a single runner, multi-tenant dev hosts), even though they have nothing to do with each other.

Source: cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp around line 611:

NixlTransferAgent::NixlTransferAgent(BaseAgentConfig const& config) : mName{config.mName}
{
    if (config.useListenThread)
    {
        FileLock lock("/tmp/trtllm_nixl_port.lock");      // <-- hardcoded
        if (!lock.lock())
            TLLM_THROW("Failed to lock /tmp/trtllm_nixl_port.lock");
        ...
    }
}

The FileLock class itself is generic — only the call site bakes in the path.

Repro context

Running two trtllm engine processes from two different Docker containers, each launched with -v /tmp:/tmp (the default for several internal toolchains), with useListenThread=true (e.g. multimodal frontend→backend transport, disagg P/D). Both containers' processes try to take /tmp/trtllm_nixl_port.lock. One wins, the other throws Failed to lock /tmp/trtllm_nixl_port.lock.

Concretely we hit this in the ai-dynamo/dynamo test suite when running tests/serve/test_trtllm.py::test_deployment[aggregated_multimodal_frontend_decoding-2] from two containers (runtime + dev image of the same SHA) in parallel on the same host. Tests that don't open a NIXL listener (the plain aggregated* cases) are unaffected.

Suggested fix

Honor an env var override for the lock path. Drop-in:

auto const* envLock = std::getenv("TRTLLM_NIXL_PORT_LOCK_PATH");
std::string lockPath = envLock && envLock[0]
    ? envLock
    : "/tmp/trtllm_nixl_port.lock";
FileLock lock(lockPath);

Or fall through to TMPDIR if it is set:

auto const* tmpdir = std::getenv("TMPDIR");
std::string lockPath = (tmpdir && tmpdir[0])
    ? std::string(tmpdir) + "/trtllm_nixl_port.lock"
    : "/tmp/trtllm_nixl_port.lock";

Either change keeps the existing default behavior, costs nothing for users not on shared-/tmp setups, and unblocks the multi-container / multi-tenant case.

Workaround today

We isolate /tmp per container (override -v /tmp:/tmp with a per-container scratch directory or --tmpfs /tmp) and serialize concurrent trtllm engine launches that need a listener. This is awkward — /tmp is shared by default in many container toolchains, and the failure mode (a literal Failed to lock /tmp/... throw on engine init) is opaque to anyone not familiar with this code.

Start in cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp around the NixlTransferAgent constructor and inspect the FileLock call used when useListenThread is enabled. Confirm the existing default behavior, then validate the configurable path with tests/serve/test_trtllm.py::test_deployment[aggregated_multimodal_frontend_decoding-2] and parallel container runs; done means unrelated processes no longer contend on the same lock when configured separately.