kvcache-ai / kvcache-ai/Mooncake

[TENT][Bug] RDMA worker self-deadlocks when endpoint retry fills BoundedMPSCQueue

Open
#3,636 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

## Summary

A failed RDMA endpoint bootstrap can deadlock all TENT RDMA workers.

This is distinct from #3604. #3604 evicts stale endpoint/QP state after RDMA port recovery; it does not protect TENT software work queues from blocking retries.

## Trigger

The affected Prefill process logged endpoint bootstrap failures such as:

```text
Unable to connect endpoint ...@mlx5_bond_7:
InvalidArgument: Missing peer GID in bootstrap
```

The RDMA link and GIDs were present and active when inspected, so this appears to be a bootstrap/endpoint-lifecycle failure rather than a currently down physical port.

## Observed deadlock

`Workers::asyncPostSend()` retries failed slices through `submit(slice)`. `Workers::submit()` calls `BoundedMPSCQueue::push()`.

`push()` has no full-queue failure or timeout path. When the queue is full it loops indefinitely with `std::this_thread::yield()`.

GDB on the affected PP2 process shows all 6 RDMA workers blocked in the same path:

```text
sched_yield
BoundedMPSCQueue::push
Workers::submit(RdmaSliceList&, int)
Workers::submit(RdmaSlice*)
Workers::asyncPostSend()
Workers::workerThread(int)
```

Therefore no worker can return to `asyncPostSend()` to consume queues or to `asyncPollCq()` to poll CQs. The queues cannot drain.

External submitter threads also block at:

```text
TransferEnginePy::batchTransferSync
TransferEngine::submitTransfer
RdmaTransport::submitTransferTasks
Workers::submit
BoundedMPSCQueue::push
```

The Python wrapper times out after 20 seconds, but the native submitter thread remains blocked. This eventually causes all Prefill requests to hit the 90-second TTFT timeout.

## Root cause

Worker-internal retry is a blocking producer operation on the same bounded queues that the worker itself must consume. Once all workers block in `push()`, the system self-deadlocks.

## Expected behavior

Endpoint/bootstrap failures should apply bounded retry/backpressure and eventually fail affected tasks. They must not block all RDMA worker threads indefinitely.

## Proposed direction

1. Add non-blocking or timed `tryPush()` to `BoundedMPSCQueue`.
2. Propagate queue-full status from `Workers::submit()`.
3. Never perform an unbounded blocking `submit()` from `asyncPostSend()`.
4. Retry grouped slice lists with backoff and a bounded retry budget.
5. On queue full or exhausted retry budget, fail affected slices/tasks instead of spinning.
6. Add a regression test that forces endpoint bootstrap failure with a small queue capacity and asserts workers continue consuming/polling while the batch fails cleanly.

## Relation to #3604

#3604 can prevent one trigger (stale QPs after a port flap). It does not address this software-queue self-deadlock, which can still be triggered by bootstrap failures such as `Missing peer GID in bootstrap`.

Contributor guide

Open the contributing guide

Research direction

Start by locating BoundedMPSCQueue::push(), Workers::submit(), asyncPostSend(), and asyncPollCq(), then trace how queue-full status can move through the RDMA worker path. Add the proposed bounded retry or backpressure behavior and a regression test using a small queue with forced endpoint bootstrap failure. Done means workers continue consuming and polling, while the affected batch fails cleanly instead of spinning indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.