Dstack-TEE / Dstack-TEE/private-ai-gateway
Allow callers to control prefix-cache partitioning
- Dominant language
- Rust
- Stars
- 32
- Forks
- 8
- Avg merge
- 11h 9m
- Merged PRs (30d)
- 36
Description
## Problem
Private AI Gateway does not provide consistent caller control over prefix-cache partitioning.
For the active Kimi-K2.6 providers:
- Chutes forwards a caller's optional `cache_salt` to vLLM.
- Tinfoil strips `cache_salt` and derives its own value. It accepts the router-only `user_cache_secret` as the caller-controlled input, but the gateway does not currently translate the public field.
As a result, the same `cache_salt` request has different behavior across providers.
## Desired behavior
Keep `cache_salt` caller-controlled across providers:
- Forward `cache_salt` unchanged to Chutes and direct vLLM providers.
- Translate `cache_salt` to `user_cache_secret` for Tinfoil, which derives the final salt.
- Do not derive, overwrite, or bind the partition to authenticated Redpill tenant identity.
- Preserve current provider behavior when the caller omits `cache_salt`.
Callers may choose distinct values for isolation or reuse a value to share a cache namespace. This is user-controlled partitioning, not a gateway-enforced tenant boundary.
## Acceptance criteria
- A supplied `cache_salt` reaches Chutes and direct vLLM providers unchanged.
- The same public field reaches Tinfoil as `user_cache_secret`.
- Omitting `cache_salt` does not add a gateway-generated partition.
- Tests cover provider-specific request transformation and receipt consistency.
- Documentation explains the caller-controlled semantics and provider translation.
## Evidence
Tinfoil's attested router strips `cache_salt`, consumes `user_cache_secret`, and derives the final salt: https://github.com/tinfoilsh/confidential-model-router/blob/v0.0.118/cache_salt.go
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing provider-specific request transformation for Chutes, direct vLLM, and Tinfoil, then compare it with Tinfoil's cache_salt.go behavior. Verify supplied and omitted cache_salt cases through provider transformation and receipt-consistency tests; done when the documentation covers caller-controlled semantics and all acceptance criteria pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100