kvcache-ai / kvcache-ai/Mooncake
[TransferEngine] TcpTransport benchmark note: VRAM-backed transfers pay per-chunk staging; original "2x host-path loss on USE_CUDA" was a benchmark-config artifact
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report / Performance
**TcpTransport throughput drops ~2x when the build enables USE_CUDA, even for pure host-memory transfers** — measured on loopback with `transfer_engine_bench` (write, 4 threads × batch 32, duration 3–4 s, same box, same commit `f1f5e40`, only the build flag differs):
| block size | build without CUDA | build with `-DUSE_CUDA=ON` |
|---|---|---|
| 64 KB | 0.99 GB/s | 0.84–0.88 GB/s |
| 256 KB | 2.10 GB/s | 1.09–1.28 GB/s |
| 1 MB | **3.41 GB/s** | **1.53–1.64 GB/s** |
All buffers are plain DRAM (`numa_alloc`/`malloc`); no GPU memory is involved in the transfer.
### Root cause (from reading `tcp_transport.cpp`)
Under `USE_CUDA` (and the other accelerator macros), both `ClientSession::writeBody`/`readBody` and `ServerSession::writeBody`/`readBody` call `getCudaDeviceId(addr)` → `cudaPointerGetAttributes()` **once per 64 KB chunk** (not once per request), on both ends of the connection. For a 1 MB transfer that is 16 chunks × 2 sides = 32 CUDA runtime calls per request; `cudaPointerGetAttributes` costs on the order of a microsecond and serializes on CUDA runtime internals, which is enough to halve throughput at these block sizes. The result of the lookup cannot change between chunks of one request — the address range is fixed at submit time.
### Suggested fix
Resolve the memory type **once per request** (at header processing / `initiate`) and carry `cuda_device` through the chunk loop, instead of re-querying per chunk. That is a small, behavior-preserving change; a further step could consult the registered-buffer metadata (the engine already knows the location string of registered memory) and skip the CUDA query entirely for buffers registered as `cpu:*`.
Happy to pick this up as a follow-up to my in-flight TCP work if nobody else wants it — the benchmark setup above reproduces it in a couple of minutes on any CUDA-build machine.
### Environment
- main @ `f1f5e40`; x86_64 Linux with a recent GCC/CUDA toolchain; loopback P2PHANDSHAKE metadata.
Contributor guide
Research direction
Start in tcp_transport.cpp at ClientSession::writeBody/readBody and ServerSession::writeBody/readBody, then run transfer_engine_bench with the loopback setup described in the issue. Verify that memory type resolution happens once per request rather than once per 64 KB chunk, and confirm the benchmark no longer shows the reported CUDA-build throughput loss for host-memory transfers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- networking, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100