[Feature] [RFC] [HiCache] Out-of-process HiCache data plane with device-memory IPC

### Checklist

- [x] If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
- [x] Please use English. Otherwise, it will be closed.

### Motivation

### Summary

I would like to discuss whether HiCache should support an **out-of-process data-plane mode**, where a local `HiCacheDaemon` runs separately from the scheduler process and accesses the scheduler's GPU KV pool through a device-memory IPC mechanism (**CUDA IPC on NVIDIA as the initial backend**).

A key design question is where the authoritative cache metadata boundary should live. For a minimal prototype, keeping `UnifiedRadixCache` / TreeCore scheduler-local may reduce scheduler-path IPC and narrow the correctness surface, while daemon-owned TreeCore could eventually enable a single authoritative cache index across multiple schedulers.

This RFC focuses first on the out-of-process data plane and the ownership boundary around it.

A conservative initial split could look like:

```text
Scheduler process
├── request scheduling
├── UnifiedRadixCache / TreeCore
├── prefix matching
├── lock / reference semantics
├── GPU KV page allocation
└── decides backup / restore intents
│
│ device-memory IPC + lightweight control messages
│ (CUDA IPC initially on NVIDIA)
▼
HiCacheDaemon
├── imported GPU KV mappings
├── GPU <-> Host transfers
├── host KV cache / physical residency
├── asynchronous transfer queues
├── prefetch / backup execution
├── write-back / write-through execution
└── external L3 storage clients
```

This would make HiCache's lower-tier cache and transfer path independently deployable while keeping scheduler-critical cache semantics local in the initial design.

This RFC is primarily about the **process and ownership boundary**. It is not a claim that the proposed architecture is necessarily faster than the current in-process implementation.

### Why consider this?

Today, HiCache transfer and lower-tier cache management are closely coupled to the scheduler process.

That has some potential drawbacks:

1. **Scheduler complexity**

GPU/host transfers, asynchronous backup/load-back, storage I/O, transfer queues, and lower-tier cache lifecycle add concurrency and I/O state to the scheduler process.

2. **Failure and lifecycle coupling**

A scheduler restart also tears down its HiCache runtime state, transfer workers, storage connections, and pending operations.

3. **Independent evolution**

A standalone data plane could evolve, be profiled, restarted, and instrumented independently from request scheduling.

4. **Transfer resource management**

A daemon could centrally own device transfer streams, pinned host memory, batching, throttling, storage connections, and I/O observability.

5. **Bulk KV data does not need to cross an RPC boundary**

With device-memory IPC, the daemon can map the scheduler's existing GPU KV allocations. Normal operations can carry page IDs, offsets, keys, and synchronization handles instead of serialized tensors.

LMCache's multiprocess mode provides a useful reference point here: LMCache can run as a standalone service, and its multiprocess connector supports a device-handle path where the cache service operates on engine KV buffers through device IPC (CUDA IPC on NVIDIA).

LMCache also already contains an SGLang multiprocess adapter that communicates with a standalone LMCache daemon, so this process model is relevant to SGLang integration today as well.

### Proposed boundary

One key design question is whether TreeCore should remain scheduler-local or become daemon-owned.

For a minimal prototype, keeping TreeCore local has a useful property: operations such as

```text
match_prefix()
insert()
inc_lock_ref()
dec_lock_ref()
scheduler cache admission decisions
```

would not require daemon IPC, and the scheduler could remain authoritative for request-visible cache semantics.

The daemon would instead execute data movement and lower-tier residency operations.

Conceptually:

```text
┌──────────────────────────┐
│ Scheduler │
│ │
request ─────────────►│ UnifiedRadixCache │
│ │ │
│ ▼ │
│ TreeCore │
│ │ │
│ ▼ │
│ GPU page IDs │
│ │ │
│ GPU KV Pool │
└────────┼─────────────────┘
│
device-memory IPC mapping
(CUDA IPC initially)
│
▼
┌──────────────────────────┐
│ HiCacheDaemon │
│ │
│ Transfer Engine │
│ ├─ D2H │
│ ├─ H2D │
│ ├─ backup │
│ └─ restore │
│ │
│ Host KV tier │
│ │ │
│ ▼ │
│ External L3 │
└──────────────────────────┘
```

This avoids putting `match_prefix()` or other scheduler hot-path tree operations behind RPC in the initial version. However, daemon-owned TreeCore remains an interesting direction if the goal expands toward authoritative cross-scheduler cache sharing.

### Device IPC data path

At initialization, a scheduler could register its GPU KV allocations with the daemon:

```text
REGISTER_KV_POOL(
instance_id,
generation,
device_id,
layout,
memory_ipc_handles
)
```

The daemon imports those allocations once.

Subsequent operations only need descriptors, for example:

```text
BACKUP(
operation_id,
source_pages,
cache_keys,
ready_fence
)

RESTORE(
operation_id,
cache_keys,
destination_pages,
completion_fence
)
```

For the initial NVIDIA implementation, CUDA IPC would provide the memory-sharing mechanism. The daemon would still submit the actual asynchronous D2H/H2D copies on its own CUDA streams.

The long-term architecture should ideally not require CUDA-specific semantics. Other accelerator runtimes expose analogous device-memory IPC mechanisms (for example, HIP IPC and Level Zero IPC), but the exact memory-sharing and synchronization capabilities differ by platform.

In particular, I think **device-memory sharing and cross-process synchronization should be treated as separate capabilities**:

```text
DeviceMemoryIPC
├── export allocation
├── import allocation
└── close mapping

DeviceSyncIPC
├── export fence/event (when supported)
├── import fence/event
└── wait/query
```

A platform may have a usable memory IPC path without an equally reliable or available event-IPC path. In that case, the data plane should be able to select another completion mechanism rather than assuming CUDA-event semantics are universal.

Device IPC itself is probably not the main architectural challenge here. The harder question is how to split authoritative ownership of GPU pages, host residency, transfer completion, eviction, and failure recovery across the scheduler/daemon boundary.

For the control and completion path, possible implementations include:

* UDS / lightweight RPC;
* shared-memory submission/completion rings;
* eventfd + shared queues;
* platform-specific device event IPC when available;
* another explicit host/device synchronization mechanism as a fallback.

A high-frequency path may eventually benefit from an NVMe-style submission/completion queue, but I do not think the exact transport needs to be decided before agreeing on the ownership model.

### Ownership

A conservative first version could use the following ownership model:

```text
Scheduler owns:
- GPU KV allocation
- GPU page allocator
- TreeCore / radix metadata
- request references and locks
- GPU-page execution lifetime
- backup / restore decisions

HiCacheDaemon owns:
- imported device-memory IPC mappings
- transfer streams
- host KV physical allocation
- pending transfer state
- transfer scheduling / batching
- L3 storage clients
- lower-tier I/O metrics
```

In this model, the scheduler explicitly publishes source pages that are safe to back up and explicitly provides destination pages for restore.

This avoids introducing a cross-process GPU page allocator in the first version.

There is an important design question around **host-tier eviction metadata**: if TreeCore remains authoritative for cache semantics while the daemon owns physical host residency, eviction and invalidation need an explicit protocol so that neither side can observe stale residency.

That seems worth discussing before choosing the exact API.

### Failure and lifecycle handling

I think lifecycle handling should be treated as a first-class part of the design.

Each scheduler registration likely needs at least:

```text
instance_id
process identity
pool generation
device identity
```

If a scheduler exits or crashes, the daemon must eventually release:

* imported device-memory IPC mappings;
* imported synchronization objects;
* pending operations;
* scheduler-specific physical residency state.

A restarted scheduler must not consume completions or page references belonging to an older generation.

Likewise, daemon failure should ideally fail closed without corrupting scheduler-local GPU cache state.

For example, a possible degraded mode could be:

```text
HiCacheDaemon unavailable
↓
temporarily disable L2/L3 backup/load
↓
continue inference with scheduler-local L1 cache
```

Whether preserving host-cache contents across scheduler restarts is safe or useful could be considered separately.

### Relationship to existing SGLang work

This seems complementary to several recent SGLang efforts.

#### #24542 — standalone HiCache L3 StorageServer

#24542 proposed a standalone storage process using UDS/gRPC for control and shared `memfd` host memory for the bulk data path.

Its central principle is similar:

```text
control plane -> descriptors
data plane -> shared memory
```

This RFC asks whether that process boundary could move one tier closer to the accelerator:

```text
#24542

SGLang HiCache
│
shared host memory
▼
Standalone L3 StorageServer

This proposal

SGLang scheduler
│
device-memory IPC
(CUDA IPC initially)
▼
HiCacheDaemon
├── Host tier
└── L3 storage
```

#### #32710 — Rust TreeCore

The merged Rust TreeCore work makes the tree/data-plane boundary more explicit.

One possible architecture is:

```text
Scheduler
├── scheduling
├── Rust TreeCore
└── GPU allocator
│
device-memory IPC
(CUDA initially)
│
▼
HiCacheDaemon
├── transfer engine
├── host tier
└── external storage
```

This could keep the latency-sensitive metadata path in-process while moving bulk cache operations out of the scheduler. A future daemon-owned TreeCore design could be considered separately if cross-scheduler authoritative metadata becomes a goal.

#### #35648 — shared KV pool with CUDA IPC

#35648 explores cross-process KV sharing with a CUDA-IPC-visible physical KV allocation.

Although its goal is different (same-GPU model replicas), it highlights many of the same systems questions:

* ownership;
* publication;
* generations;
* reference lifetime;
* synchronization;
* failure recovery.

This proposal focuses specifically on the HiCache transfer/data plane rather than cross-scheduler shared prefix ownership.

### TreeCore placement

There are at least two plausible designs:

```text
A. Scheduler-local TreeCore

Scheduler
├── TreeCore
├── GPU allocator
└── execution
│
▼
HiCacheDaemon
├── host tier
├── transfer
└── L3
```

This minimizes scheduler-path IPC and keeps request/cache coordination in one process.

A more aggressive design is:

```text
B. Daemon-owned TreeCore

Scheduler A ─┐
Scheduler B ─┼──► HiCacheDaemon
Scheduler C ─┘ ├── TreeCore
├── cache directory
├── residency
└── transfer
```

This could provide a single authoritative cache index across multiple schedulers, but it would also move prefix matching, locking, insertion, and potentially eviction decisions across an IPC boundary and turn more of request/cache coordination into a distributed state machine.

For a first experiment, scheduler-local TreeCore seems like the smaller correctness surface. I would still like maintainer feedback on whether that is the right long-term boundary.

### Potential benefits

If this architecture works well, possible benefits include:

* less cache I/O and transfer concurrency inside scheduler processes;
* independent HiCache lifecycle and observability;
* centralized transfer batching and bandwidth control;
* cleaner storage-backend integration;
* easier experimentation with alternative data-plane implementations;
* isolation of storage or transfer failures from scheduler logic;
* a clearer separation between cache semantics and cache movement;
* potentially reusable local cache infrastructure across inference processes.

### Costs and risks

There are also meaningful costs:

* IPC/control-message overhead;
* device-memory IPC mapping and synchronization lifecycle complexity;
* harder crash cleanup;
* host-residency metadata synchronization;
* daemon accelerator-context overhead;
* additional deployment complexity;
* possible synchronization stalls if the protocol is poorly designed;
* hardware/platform differences in memory IPC and inter-process event support.

LMCache MP's experience also suggests that stale IPC mappings, worker death, event lifetime, and cleanup behavior need explicit design and testing rather than being treated as edge cases.

### Possible minimal prototype

Instead of moving all HiCache functionality at once, a useful first experiment might only support:

* NVIDIA CUDA / Linux;
* one scheduler process;
* one local daemon;
* one GPU;
* Full attention only;
* GPU <-> host backup and restore;
* CUDA IPC as the initial device-memory IPC backend;
* scheduler-owned GPU allocator;
* scheduler-local TreeCore;
* no external L3 initially.

The goal would be to compare:

```text
current in-process HiCache transfer path

vs.

scheduler
│
CUDA IPC
▼
standalone HiCacheDaemon
```

while keeping cache semantics unchanged.

Useful measurements would include:

* TTFT;
* ITL impact;
* backup/load latency;
* scheduler CPU time;
* transfer throughput;
* overlap with model execution;
* IPC/control overhead;
* GPU/host memory overhead;
* behavior under scheduler and daemon failure.

### Questions for maintainers

I would especially appreciate feedback on the following:

1. Does an out-of-process HiCache **data plane** look like a useful architectural direction for SGLang?

2. Should `UnifiedRadixCache` / TreeCore remain scheduler-local initially, or should daemon-owned TreeCore be considered from the start?

3. Should the scheduler continue to own the GPU KV allocator, with the daemon only importing its allocations through a device-memory IPC backend (CUDA IPC on NVIDIA initially)?

4. Should the host KV pool and host-tier allocator move into the daemon, or remain scheduler-owned initially?

5. What should be authoritative for host-tier residency / eviction metadata?

6. Should device-memory sharing and cross-process synchronization be modeled as separate platform capabilities, rather than assuming CUDA IPC + CUDA event IPC semantics?

7. Should one daemon serve one scheduler, one GPU, or potentially multiple scheduler processes on the same node?

8. Would an NVIDIA GPU <-> host-only prototype be a useful first step before integrating other device IPC backends and existing HiCache L3 backends?

The intent of this RFC is to discuss the **process boundary, ownership model, portability boundary, and failure semantics first**, rather than proposing an immediate large HiCache refactor.

---

### Related resources

* SGLang #24542 — `[RFC] [HiCache] A new backend for external L3 storage systems`
https://github.com/sgl-project/sglang/issues/24542

* SGLang #32710 — `[Radix Cache] Add Rust TreeCore backend with shared parity tests`
https://github.com/sgl-project/sglang/pull/32710

* SGLang #35648 — `[Feature] [RFC] Same-GPU model replicas with managed CUDA MPS and a shared KV pool`
https://github.com/sgl-project/sglang/issues/35648

* LMCache multiprocess mode / `LMCacheMPConnector`
https://github.com/LMCache/LMCache/blob/dev/docs/source/getting_started/quickstart.rst

* LMCache SGLang multiprocess adapter
https://github.com/LMCache/LMCache/blob/dev/lmcache/integration/sglang/multi_process_adapter.py

* AMD HIP IPC APIs (`hipIpcGetMemHandle`, `hipIpcOpenMemHandle`, event IPC)
https://rocm.docs.amd.com/projects/HIP/en/latest/doxygen/html/group___device.html

* Intel Level Zero IPC memory handles
https://oneapi-src.github.io/level-zero-spec/level-zero/latest/core/EXT_IPC_MEM_HANDLE_TYPE.html