[RDMA] Send/recv buffers share one registered memory pool → silent stall risk at large scale
- Dominant language
- C++
- Stars
- 17.6k
- Forks
- 4.1k
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 69
Description
## Summary
In bRPC RDMA, **all connections' send-serialization buffers and receive pre-posted buffers come from a single process-wide registered memory pool** (`AllocBlock`, `block_pool.cpp:426`), pinned via `ibv_reg_mr`. Unlike TCP (independent, pageable, kernel-managed per-socket buffers), this creates a **structural send/recv coupling** that can cause a **silent, whole-pool stall** under large-scale, mixed-load, under-provisioned deployments.
## Root cause (with code pointers)
- Global IOBuf allocator is replaced by the registered pool: `rdma_helper.cpp:578`.
- Send buffer is released only after the peer ACKs (IMM): `rdma_endpoint.cpp:965`. The peer can only receive if it has **already posted a recv buffer** — which also comes from the same pool.
- On pool exhaustion, `PostRecv(1)` fails (`rdma_endpoint.cpp:983`, `AllocBlockFrom` returns NULL at `block_pool.cpp:374`).
- Cycle: pool exhausted → recv re-post fails → no one can receive → in-flight sends never complete → ACK never returns → send buffers (same pool) never freed → pool never recovers → **all connections sharing the pool stall**.
- The per-connection credit window (`_window_size`) is congestion control only and is **independent** of `AllocBlock`; it provides no memory isolation.
## When it happens (scenarios)
Three conditions together:
1. **Mixed load** — large messages (64KB/2MB blocks) and small RPCs (8KB blocks) share one pool.
2. **High fan-out** — many concurrent connections under load.
3. **Under-provisioned pool** — especially since, by **default `buckets=4`**, the 8KB pool (`block_type 0`, carrying all normal RPC send/recv buffers) **cannot grow at runtime** (`CanExtendBlockRuntime`, `block_pool.cpp:139-141` requires `buckets==1` or `region_num<1`; `InitBlockPool` already set `region_num[0]=1`). It is locked at the initial 1GB.
Typical deployments at risk: gateways/proxies mixing big transfers and small control RPCs, multi-tenant access layers, long-lived large-message streams.
## Impact
- **Whole-pool stall, not single-request failure.** All RDMA connections sharing the pool hang simultaneously (availability-level).
- **Silent and hard to diagnose.** Only occasional `Fail to extend region` / `Fail to allocate rbuf` in logs — no "shared-pool deadlock" signal. Easy to misattribute to network/kernel (#3202 showed `ibv_post_send: Cannot allocate memory`, rooted in kernel SQ memory, while the block pool had already triggered extend).
- **No per-connection fairness/isolation.** `AllocBlockFrom` is LIFO + per-thread TLS hoarding + random bucket, biasing toward recently active (large-flow) threads.
## Questions for the community
1. Is the send/recv shared-pool coupling at scale considered expected behavior, or a real reliability risk?
2. Should docs state explicit RDMA **applicable / not-applicable** boundaries for large-scale mixed load?
3. Should `CanExtendBlockRuntime`'s `buckets==1` restriction be relaxed for `block_type 0` so the 8KB pool can grow under default config?
4. Should a per-direction (at least receive-side) reserved/guaranteed memory mechanism be introduced to avoid recv refill being starved by sends?
5. Should docs (`max_regions` default 16 vs code 3; `buckets` restriction) be aligned with code?
*Static source analysis only; recommend a repro with mixed large/small flows + high fan-out + gradually shrinking pool.*
Contributor guide
Research direction
Read rdma_helper.cpp:578, rdma_endpoint.cpp:965 and :983, and block_pool.cpp:374, :426, and :139-141 to trace allocation, receive re-posting, and pool extension. Run the proposed mixed large/small-flow, high-fan-out reproduction with a gradually shrinking pool. Done means establishing whether the stall is reproducible and getting a maintainer decision on code changes or documentation boundaries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100