[RFC]: Workload-Aware KV Cache Hints for Mooncake: `session_id` and `priority`

### Changes proposed

## Summary

This RFC proposes a first-stage workload-aware KV Cache interface for Mooncake.

The first stage only introduces two optional external hints:

- `session_id`
- `priority`

The goal is to let upper layers such as SGLang, vLLM, or llm-d pass lightweight workload information to Mooncake without changing the correctness semantics of existing `PUT` / `GET` operations.

If these hints are not provided, Mooncake behavior should remain unchanged.

## Motivation

Agentic and multi-turn workloads often reuse long prefixes across requests. However, the KV Cache objects generated by these workloads are not all equally valuable:

- KV objects belonging to the same session are often related in lifecycle and reuse pattern.
- Some requests are more latency-sensitive or more valuable to retain than others.
- Upper layers already have partial knowledge about session identity and request priority.

Mooncake currently provides object-level KV storage, lookup, transfer, replication, and eviction. To support workload-aware KV Cache management, Mooncake should be able to consume a small amount of external context while still keeping policy decisions inside Mooncake.

This RFC intentionally starts with the smallest useful surface: `session_id` and `priority`.

## Background

Recent SGLang main already exposes both signals:

- `session_id` is available as a top-level request field in SGLang's internal request path and OpenAI-compatible serving path.
- `priority` is available as an integer request field and is already used by SGLang scheduling / radix-cache logic.

SGLang also already has a Mooncake HiCache path that can pass `ReplicateConfig.group_ids`. Those `group_ids` currently represent page-level physical grouping, such as K/V objects, split-head shards, MLA objects, and sidecar objects derived from the same logical HiCache page.

This RFC does not propose replacing that existing `group_ids` behavior.

## Goals

1. Add optional `session_id` and `priority` hints to Mooncake object-level write paths.
2. Preserve existing behavior when the hints are absent.
3. Keep existing `group_ids` semantics unchanged.
4. Allow Mooncake to store session/priority metadata for KV objects.
5. Allow Mooncake to use priority as a coarse retention hint.
6. Provide a foundation for future session-level statistics, lifecycle handling, and eviction policy improvements.

## Non-Goals

This RFC does not propose:

- A new agent hint standard.
- Changes to OpenAI-compatible API standards.
- Changing `GET` semantics.
- Treating `session_id` as a cache key namespace.
- Treating `session_id` as hard pinning.
- Treating `priority` as a lease.
- Replacing existing `group_ids`.
- Adding placement hints such as `preferred_segment`.
- Adding replica hints such as `replica_num`.
- Adding explicit movement APIs such as prefetch/copy/move/demote.

Those can be discussed in later phases.

## Proposed API Semantics

### `session_id`

`session_id` is an optional string hint.

Semantics:

- It identifies the session or workload that caused the KV object to be written.
- It is metadata associated with the object.
- It should not be part of the object key by default.
- It should not prevent eviction by itself.
- It should not override `group_ids`.

Expected usage:

- Associate KV objects with sessions.
- Support session-level observability.
- Enable future session-level lifecycle operations, such as bulk release, demotion, or statistics.
- Help upper layers correlate Mooncake cache behavior with request/session routing.

Important distinction:

- `group_ids` groups related physical objects derived from the same logical page.
- `session_id` identifies the workload/session that produced or used the object.

These are orthogonal and should both be preserved.

### `priority`

`priority` is an optional integer hint.

Mooncake should treat it as a cache retention hint, not as a scheduler priority.

Recommended Mooncake-side semantics:

- Higher normalized priority means the object is more valuable to retain.
- Default priority should behave exactly like today's behavior.
- Priority should influence eviction only as a soft preference.
- Priority should not directly change lease duration.
- Priority should not imply hard pinning.

Because different upper layers may define request priority differently, the integration layer should normalize priority before passing it to Mooncake when needed.

For example, SGLang uses integer priority, but its scheduling direction may be configured. Therefore, Mooncake should not need to understand SGLang's scheduling flags. The SGLang-Mooncake adapter should map SGLang request priority into Mooncake's cache-retention priority.

## Proposed Data Model

The exact implementation should follow Mooncake's current coding style and binding constraints. Conceptually, the write configuration should support:

```cpp
struct ReplicateConfig {
// Existing fields.
// ...

// Optional workload/session identity.
// Empty string means unset.
std::string session_id;

// Optional cache-retention priority.
// 0 means default/normal behavior.
int32_t priority = 0;
};
```

If Mooncake already has a better internal optional-field convention, we should follow it.

For `batch_put`, the minimal first-stage behavior can treat `session_id` and `priority` as applying to all objects in the batch.

If a caller needs to write objects from multiple sessions or priority classes in the same batch, it can split the batch in the first stage. If this becomes a performance issue, we can add per-object vectors later, similar to the existing `group_ids` pattern:

```cpp
std::vector session_ids;
std::vector priorities;
```

This RFC does not require per-object vectors in the first implementation.

## Expected Mooncake Behavior

### When hints are absent

Behavior must be unchanged.

Existing clients, tests, and SGLang integrations should continue to work without modification.

### When `session_id` is present

Mooncake should record the session association in object metadata where feasible.

For the first implementation, this can be metadata-only. It does not have to immediately change eviction behavior.

If an object already exists and a new `PUT` is skipped or deduplicated, there are two possible acceptable first-stage behaviors:

1. Best-effort metadata update: record that this session also touched the object.
2. Insert-only metadata: keep only the metadata from the original write.

Option 1 is more useful for session-level accounting, but option 2 is acceptable for an initial minimal implementation if updating metadata for existing objects is invasive. The chosen behavior should be documented.

### When `priority` is present

Mooncake should store the priority in object metadata.

If the current eviction path can consume object priority with a small change, Mooncake should use it as a soft eviction preference. For example:

- normal priority: existing behavior
- high priority: evict later than normal objects when possible

If the current eviction path does not have a clean priority hook, the first stage may only store and expose priority metadata, then add eviction integration in a follow-up PR.

Priority should be normalized into a small number of internal levels at first. Mooncake should not depend on arbitrary raw integer values from upper layers.

## Interaction With Existing `group_ids`

Existing `group_ids` behavior must remain unchanged.

For example, if SGLang writes physical objects derived from the same logical HiCache page:

```text
page0_k
page0_v
page0_sidecar
```

they may share the same `group_ids` value.

If the request also has `session_id = "session-A"`, then both pieces of information should coexist:

```text
object key: page0_k
group_id: sglang-hicache:page0
session_id: session-A
priority: normal/high
```

`session_id` must not overwrite `group_ids`.

## Integration With SGLang

The initial SGLang-Mooncake integration can map:

- SGLang `session_id` -> Mooncake `session_id`
- SGLang request priority -> normalized Mooncake `priority`

Other SGLang signals should be left for later phases:

- `routed_dp_rank`
- `disagg_prefill_dp_rank`
- `routing_key`
- `extra_key` / `cache_salt`
- `cached_tokens_details`
- KV events

For placement, the first stage can keep the existing local-first `PUT` behavior. Mooncake does not need `routed_dp_rank` or `disagg_prefill_dp_rank` for correctness.

## Compatibility

This proposal must be backward-compatible:

- Existing clients that do not set `session_id` or `priority` should behave exactly the same.
- Existing `ReplicateConfig.group_ids` users should continue to work.
- Existing Python bindings should accept old call patterns.
- Mixed-version behavior should degrade gracefully when the installed Mooncake package does not expose the new fields.

## Testing Plan

The implementation should add tests covering:

1. Existing `put` / `batch_put` behavior without hints.
2. `put` with `session_id`.
3. `batch_put` with `session_id`.
4. `put` with `priority`.
5. `batch_put` with `priority`.
6. `group_ids` and `session_id` being present at the same time.
7. `group_ids` not being overwritten by `session_id`.
8. Default priority behaving the same as current behavior.
9. Priority metadata being observable or consumed by eviction logic, depending on the implementation stage.

## Rollout Plan

Suggested implementation stages:

### Stage 1: API and metadata

- Add `session_id` and `priority` to the write config path.
- Preserve default behavior when unset.
- Store the metadata internally.
- Add tests and documentation.

### Stage 2: Observability

- Expose basic session/priority statistics if practical.
- Examples:
- bytes written by session
- objects written by session
- evictions by priority class

### Stage 3: Eviction integration

- Use normalized priority as a soft eviction preference.
- Keep priority coarse-grained initially.
- Avoid lease changes and hard pinning unless explicitly requested by a separate policy.

## Open Questions

1. Should `batch_put` support only scalar `session_id` / `priority` in the first stage, or should it immediately support per-object vectors?
2. When a `PUT` finds that the object already exists, should Mooncake update session metadata for the existing object?
3. What internal priority levels should Mooncake use initially?
4. Should priority influence eviction in the first PR, or should the first PR only add metadata and API support?
5. What observability signals are most useful for llm-d/SGLang routing feedback?

## Proposed First PR Scope

For the first PR, I suggest keeping the scope narrow:

- Add optional `session_id` and `priority` to the object write config.
- Preserve existing behavior when unset.
- Keep `group_ids` unchanged.
- Store metadata for inserted objects.
- Add tests for compatibility and metadata propagation.
- If there is a clean eviction hook, map priority into a coarse soft-retention level; otherwise defer eviction changes to a follow-up PR.

This gives Mooncake a concrete first step toward workload-aware KV Cache without requiring new agent-level APIs or large policy changes.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Start by locating the ReplicateConfig write path, including put and batch_put, the Python bindings, and the existing group_ids handling. Review the current compatibility tests and eviction path before deciding how session_id and priority should be represented. Done means backward-compatible optional fields, preserved group_ids behavior, stored metadata, and tests covering the listed write and compatibility cases.

Written by the indexing model from the issue text.