LMCache / LMCache/LMCache

[Feature][MP] Add io_uring fixed-buffer support for Lazy and Mixed L1 allocators

Open
#4,100 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.9k
Forks
1.9k
Avg merge
4d 4h
Merged PRs (30d)
141

Description

**Label**

new feature

**Is your feature request related to a problem? Please describe.**

io_uring fixed-buffer registration is not implemented for the MP `raw_block` path. There was an initial attempt in [#3246](https://github.com/LMCache/LMCache/pull/3246), but MP requires a different integration from the legacy `RustRawBlockBackend`.

The legacy backend owns its CPU backend and can register allocator buffers directly during initialization. In MP mode:

- `RawBlockL2Adapter` does not own the L1 allocator.
- L1 memory is managed separately by `L1MemoryManager`.
- MP uses either `MixedMemoryAllocator` or `LazyMemoryAllocator`.
- Lazy memory is pinned and published incrementally.

Both allocators also have a boundary problem. io_uring limits each registered buffer to 1 GiB, while both allocators manage a larger contiguous address space. `AddressManager` may therefore return an allocation spanning two registered-buffer indices. One `ReadFixed` or `WriteFixed` SQE cannot handle such a request because it contains only one buffer index.

**Describe the solution you'd like**

Add an MP-specific interface for exposing fixed-buffer regions and notifying `RawBlockL2Adapter` when new regions become available.

For both allocators:

1. Divide eligible L1 memory into regions no larger than 1 GiB.
2. Assign one fixed-buffer index to each region.
3. Change Rust lookup from exact base-pointer matching to range containment.
4. Split an I/O that crosses region boundaries into multiple fixed SQEs, one per registered region.
5. Advance the buffer pointer, length, and device offset for each segment.
6. Aggregate the segment completions into one logical I/O completion.
7. Fall back to ordinary io_uring I/O only when registration is unavailable or part of the request is not covered by registered regions.

### MixedMemoryAllocator

Mixed owns its complete arena at initialization, so all eligible regions can be registered once before I/O starts.

Its `BINARY_BUFFER` allocations and unsupported backing types should continue using ordinary I/O. In particular, POSIX shared-memory backing may not be eligible for io_uring fixed-buffer registration and must fall back cleanly.

### LazyMemoryAllocator

Lazy should use the sparse-buffer APIs available since Linux 5.13:

1. Reserve a sparse table based on the final L1 size.
2. Register the initially committed regions before serving requests.
3. For each expansion:
1. Pin the new host-memory range for CUDA.
2. Add it to the io_uring fixed-buffer table.
3. Publish it through `AddressManager.sbrk()`.
4. Unregister fixed buffers before unpinning or releasing the memory.

The registration interface must avoid races with the background expansion thread, for example through an atomic snapshot-and-subscribe operation.

The default lazy initial size is 20 GiB, so the initial prefix requires multiple 1 GiB fixed-buffer indices rather than a single index.

**Describe alternatives you've considered**

One option is to make `AddressManager` region-aware so allocations cannot cross 1 GiB boundaries. However, introducing separate address managers or changing the existing free-list behavior would be difficult and invasive. It would affect allocation, batched allocation, coalescing, fragmentation, and allocations larger than one region.

Instead, splitting crossing I/O in the Rust raw-block layer appears to be a better fit. It keeps the existing allocators unchanged and works for both `MixedMemoryAllocator` and `LazyMemoryAllocator`.

Each crossing request can be divided at registered-region boundaries, with the buffer pointer and device offset advanced for each segment. Every segment uses its corresponding fixed-buffer index, and their completions are aggregated into one logical result.

This adds completion and partial-failure handling in Rust, but keeps fixed-buffer boundary handling localized to the I/O layer where the restriction originates.

**Summary**

- Add an MP-specific interface for exposing fixed-buffer regions.
- Register Mixed regions once and Lazy regions incrementally.
- Divide L1 memory into regions no larger than 1 GiB.
- Split crossing requests into multiple fixed SQEs in Rust.
- Aggregate all segments into one logical completion.
- Fall back to ordinary io_uring when fixed-buffer registration cannot be used.

Contributor guide

Open the contributing guide

Research direction

Start with the Rust raw-block layer, RawBlockL2Adapter, AddressManager, MixedMemoryAllocator, and LazyMemoryAllocator to trace fixed-buffer registration and allocation publication. Done means MP regions are registered within the 1 GiB limit, crossing I/O is split and recombined, Lazy expansions are race-safe, and unsupported or uncovered requests fall back to ordinary I/O.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.