handle_request() occasionally hangs forever (not a GPU fault) -- confirmed live with py-spy + gdb, no Xid errors

Summary

LLM._runtime.handle_request(request) (the pybind entry point into the compiled C++ engine, called from the experimental server's chat_completions() handler) occasionally never returns -- a genuine hang inside the native runtime, not a GPU crash, not a Python-level issue, and not related to the separate idle-scheduler issue reported in #180.

Caught this live in production with py-spy dump (Python-symbol-resolved stack + locals) at the exact moment of a hang. The stuck thread is sitting at:

Thread N (active): "AnyIO worker thread"
    chat_completions (experimental/server/api_server.py:217)
        response = llm_instance._runtime.handle_request(request)

The Python-side request is fully constructed at the point of the hang (SamplingParams, LLMGenerationRequest, tool_config all built; temperature=0, top_p=1, top_k=1 -- i.e. this reproduces independently of the sampling-params issue in #180). Execution has entered the pybind call and never returns.

Why we're confident this is NOT a GPU-level fault

• dmesg shows zero Xid errors (the standard NVIDIA kernel-driver signal for a real GPU crash/fault) across every hang window we checked.
• gdb -batch -ex "thread apply all bt" on the hung process at the same moment shows the two CUDA driver background threads (cuda00001400006, cuda-EvtHandlr) sitting in poll() inside libcuda.so.1 -- their normal idle-polling state, not stuck inside a kernel launch or CUDA API call.
• The main asyncio event loop thread is alive and idling normally in epoll_pwait.
• numpy's OpenBLAS thread pool threads are idling normally in pthread_cond_wait.
• Overall container CPU usage at hang time was low (5-49% across two captures, consistent with other real requests/idle background threads, not a busy-spin).

Everything else in the process looks completely normal; only the one thread that entered handle_request() is stuck, and it never comes back.

Environment

• TensorRT-Edge-LLM: v0.9.1 (7f061f2, tag v0.9.1)
• Hardware: Jetson AGX Orin (unified memory)
• TRT_VERSION: 10.16.1.11
• Model: Qwen3.6-35B-A3B GPTQ-Int4 (prebuilt engine, max_batch_size=1)
• Server: the experimental OpenAI-compatible server (experimental.server), run via a small wrapper script that adds a request-serialization semaphore in front of /v1/chat/completions (necessary at max_batch_size=1 -- see #151/#179 for the related decode-slot-corruption history)

Observed behavior in production

Running as a long-lived service with real (not synthetic-benchmark) traffic, this hang recurs roughly every 6-40 minutes under normal load. Once it happens, the engine is completely unresponsive on /v1/chat/completions (though /health and /v1/models, which don't call into the engine, stay responsive) until the container is killed and restarted -- there is no in-process recovery. We've worked around this operationally with a healthcheck tuned to only fail after a hang exceeds our own client-side completion timeout (240s), so we don't fight it, we just restart -- but the underlying hang itself is what we're reporting here.

We do not have access to handle_request()'s C++ source (it ships as a compiled .so), so we can't point to a specific lock or code path inside it -- only that Python-level tracing conclusively shows the call itself is what never returns, under conditions that rule out a GPU fault.

Ask

1. Is handle_request() known to have any internal blocking/lock-acquisition path that can wait indefinitely under some request or engine-state condition (e.g. KV-cache eviction, a scheduler queue, spec-decode bookkeeping)?
2. Is there a recommended way to make this call itself time-bounded from the Python side (an internal cancellation/interrupt mechanism), rather than relying entirely on an external process killing the whole engine?
3. Any diagnostic flags/env vars we could enable on a future occurrence to get more visibility from the C++ side (verbose scheduler logging, a lock-contention trace, etc.)?

Happy to provide more captures (we now have py-spy reliably attached and ready) or test candidate diagnostics against our Jetson AGX Orin setup.

Start at experimental/server/api_server.py:217 and trace the pybind call to LLM._runtime.handle_request(request). Review the py-spy and gdb observations, then reproduce the hang on the stated Jetson AGX Orin environment with the available diagnostics. Done means identifying the native blocking path or documenting a supported timeout, cancellation mechanism, or diagnostic configuration.