kvcache-ai / kvcache-ai/Mooncake

[Bug]: NoF client cannot handle unaligned buffer lengths and addresses

Open
#4,029 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Bug Report

Related to [#4021](https://github.com/kvcache-ai/Mooncake/issues/4021) and [PR #4024](https://github.com/kvcache-ai/Mooncake/pull/4024).

#4024 addresses master-side disk allocation alignment and reserves padding space for objects whose sizes are not multiples of the namespace block size. For example, with `block_size=512 B`, a 513 B object receives at least 1024 B of disk space, including the 511 B needed to complete the last LBA. Its logical size remains 513 B.

This additional space is reserved on disk; it does not extend the caller's memory buffer. The client therefore needs to construct a 1024 B I/O without reading or writing beyond the caller's 513 B buffer.

Upstream applications such as LMCache and SGLang pass logical object sizes, which are not necessarily multiples of the namespace `block_size`. The actual buffer address may also be unaligned. Currently, the client rejects these requests when its alignment checks fail.

The client should adapt I/O lengths and buffer layouts while preserving logical object sizes, minimizing payload copies, and allowing `memcpy` when necessary.

**The RDMA path should use SGLs to describe the data.** The NVMe base specification distinguishes the following SGL capabilities:

| Support level | Meaning |
|---|---|
| SGL unsupported | The corresponding SGL data-description path is unavailable |
| SGL supported with DWORD alignment | Data-segment addresses and lengths must be 4-byte aligned |
| Byte-granular SGL supported | No additional alignment requirements for data-segment addresses or lengths |

The client should select a strategy based on the usable transport capabilities, buffer layout, and memory registration, rather than SGL support alone:

- **Direct SGL transfer:** Describe the original buffer and separate padding. Use zero padding for writes and a separate writable padding buffer for reads, without copying valid payload data.
- **Tail-only copy:** When the complete-LBA prefix can be transferred directly, copy only the remaining valid bytes into a temporary tail block. For reads, copy only the valid tail bytes back after completion.
- **Full payload copy:** When the prefix cannot be transferred directly because of its address, layout, or registration, use a suitable staging buffer. Process the object in chunks if needed to bound temporary memory usage.

For example, if the first 512 B of a 513 B object can be transferred directly, only the final byte needs to be copied into a zero-padded 512 B tail block. The prefix and tail block can form a valid SGL or be submitted separately.

A tail-only copy cannot fix an incompatible starting address. If the selected path requires 4-byte alignment but the buffer starts at `base + 1`, full staging is needed. A byte-granular path should not introduce copies solely because the address is not 4-byte aligned.

**The TCP path should use local iovecs to pad the I/O length.** SPDK `readv/writev` can describe:

- Writes: original data plus separate zero padding.
- Reads: the original destination plus a separate padding buffer.

The total I/O length should satisfy LBA alignment while the logical object size remains unchanged. Callers should not need to extend their buffers. Prefer direct use of the original buffer, with temporary buffers and necessary copies when direct use is unavailable.

Expected results:

- Support valid inputs with non-block-multiple lengths and unaligned memory addresses on namespaces with 512 B and 4096 B block sizes, without accessing memory beyond the caller's buffer.
- Preserve correct readback data, logical object sizes, returned lengths, and checksums.
- Cover single, batch, and multi-buffer operations without adding unnecessary payload copies to requests that can already be transferred directly.

### Before submitting...

- [x] Ensure you searched for relevant issues and read the [documentation]

Contributor guide

Open the contributing guide

Research direction

Start by locating the NoF client alignment checks and its RDMA and TCP I/O paths, including the SPDK readv/writev calls. Trace single, batch, and multi-buffer operations for 512 B and 4096 B block sizes. Done means unaligned buffers and non-block-multiple logical sizes work without out-of-bounds access, while readback data, lengths, checksums, and copy minimization remain correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
networking, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.