Dstack-TEE / Dstack-TEE/private-ai-gateway

Allow callers to control prefix-cache partitioning

Open
#83 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32
Forks
8
Avg merge
11h 9m
Merged PRs (30d)
36

Description

## Problem

Private AI Gateway does not provide consistent caller control over prefix-cache partitioning.

For the active Kimi-K2.6 providers:

- Chutes forwards a caller's optional `cache_salt` to vLLM.
- Tinfoil strips `cache_salt` and derives its own value. It accepts the router-only `user_cache_secret` as the caller-controlled input, but the gateway does not currently translate the public field.

As a result, the same `cache_salt` request has different behavior across providers.

## Desired behavior

Keep `cache_salt` caller-controlled across providers:

- Forward `cache_salt` unchanged to Chutes and direct vLLM providers.
- Translate `cache_salt` to `user_cache_secret` for Tinfoil, which derives the final salt.
- Do not derive, overwrite, or bind the partition to authenticated Redpill tenant identity.
- Preserve current provider behavior when the caller omits `cache_salt`.

Callers may choose distinct values for isolation or reuse a value to share a cache namespace. This is user-controlled partitioning, not a gateway-enforced tenant boundary.

## Acceptance criteria

- A supplied `cache_salt` reaches Chutes and direct vLLM providers unchanged.
- The same public field reaches Tinfoil as `user_cache_secret`.
- Omitting `cache_salt` does not add a gateway-generated partition.
- Tests cover provider-specific request transformation and receipt consistency.
- Documentation explains the caller-controlled semantics and provider translation.

## Evidence

Tinfoil's attested router strips `cache_salt`, consumes `user_cache_secret`, and derives the final salt: https://github.com/tinfoilsh/confidential-model-router/blob/v0.0.118/cache_salt.go

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing provider-specific request transformation for Chutes, direct vLLM, and Tinfoil, then compare it with Tinfoil's cache_salt.go behavior. Verify supplied and omitted cache_salt cases through provider transformation and receipt-consistency tests; done when the documentation covers caller-controlled semantics and all acceptance criteria pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.