kvcache-ai / kvcache-ai/AgentENV
Cache sandbox-to-node routing in the AENV gateway
- Dominant language
- Rust
- Stars
- 3.5k
- Forks
- 309
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 66
Description
## Affected component
Gateway / scheduler
## Problem statement
The gateway calls the scheduler `LookupNode` RPC for every request containing a sandbox ID. This adds scheduler load and latency.
## Use case
Large scale deployments handling frequent data-plane and control-plane requests for long-lived sandboxes. The same sandbox may receive many requests through different gateway replicas, while deletes and node changes may be handled by any replica.
## Current behavior and workarounds
Every sandbox-routed request performs a scheduler lookup. There is no gateway cache. The scheduler has no explicit binding-delete RPC, and `ReportSandboxEvent` currently logs delete events without mutating the binding store. A local-only delete eviction would therefore leave other gateways with stale entries until their process restarts or a scheduler binding expires.
## Desired behavior
- Repeated requests for an active sandbox should avoid scheduler lookups while a valid route-cache entry exists.
- Each gateway may keep a bounded, process-local positive TTL cache keyed by sandbox ID and storing the node ID and endpoint.
- The scheduler remains the source of truth. Successful lookups may be cached; `NotFound` and scheduler errors must not be negative-cached.
- After a successful delete, the handling gateway evicts its local entry and the shared scheduler binding is removed, so other gateways converge after cache expiry or stale-route detection.
- If a cached route receives a reliable sandbox-not-found response, the gateway evicts the entry and performs one uncached scheduler lookup. If the scheduler also returns `NotFound`, return 404. If it returns a new node, retry once only for a replay-safe request.
- Streaming, WebSocket, non-replayable, and ambiguous data-plane 404 requests must not be retried.
- Cache hits, misses, evictions, and refetches should be observable, and concurrent misses for one sandbox should be coalesced.
## Proposed approach
Add a bounded process-local cache in the gateway. Query the configured scheduler on cache misses, and populate or refresh entries after successful `LookupNode` calls and successful create or fork assignment recording.
## Compatibility and operational impact
API/config changes:
No client-facing API change is required.
Snapshot or storage format changes:
None.
New host/runtime requirements:
None. The cache is in process memory.
## Alternatives considered
- Keep doing a scheduler lookup for every request: correct but adds avoidable scheduler traffic and latency.
- Use a distributed cache or pub/sub invalidation for every gateway: provides faster invalidation but adds infrastructure and another consistency path.
- Evict only the gateway that handled delete: insufficient because other gateways retain stale mappings.
## Pre-submission checklist
- [x] I searched existing issues and discussions and did not find a duplicate.
- [x] I described a concrete problem and use case, not only a proposed technology.
- [x] I understand that acceptance of the problem does not imply approval of a particular implementation.
Contributor guide
Research direction
Start by tracing the gateway request path that calls the scheduler LookupNode RPC, then inspect ReportSandboxEvent handling and the shared binding store behavior. Done means valid positive routes are cached and coalesced, required evictions and safe refetches work across the stated request types, and cache activity is observable without negative caching.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100