kvcache-ai / kvcache-ai/AgentENV

Cache sandbox-to-node routing in the AENV gateway

Open
#204 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.5k
Forks
309
Avg merge
1d 9h
Merged PRs (30d)
66

Description

## Affected component

Gateway / scheduler

## Problem statement

The gateway calls the scheduler `LookupNode` RPC for every request containing a sandbox ID. This adds scheduler load and latency.

## Use case

Large scale deployments handling frequent data-plane and control-plane requests for long-lived sandboxes. The same sandbox may receive many requests through different gateway replicas, while deletes and node changes may be handled by any replica.

## Current behavior and workarounds

Every sandbox-routed request performs a scheduler lookup. There is no gateway cache. The scheduler has no explicit binding-delete RPC, and `ReportSandboxEvent` currently logs delete events without mutating the binding store. A local-only delete eviction would therefore leave other gateways with stale entries until their process restarts or a scheduler binding expires.

## Desired behavior

- Repeated requests for an active sandbox should avoid scheduler lookups while a valid route-cache entry exists.
- Each gateway may keep a bounded, process-local positive TTL cache keyed by sandbox ID and storing the node ID and endpoint.
- The scheduler remains the source of truth. Successful lookups may be cached; `NotFound` and scheduler errors must not be negative-cached.
- After a successful delete, the handling gateway evicts its local entry and the shared scheduler binding is removed, so other gateways converge after cache expiry or stale-route detection.
- If a cached route receives a reliable sandbox-not-found response, the gateway evicts the entry and performs one uncached scheduler lookup. If the scheduler also returns `NotFound`, return 404. If it returns a new node, retry once only for a replay-safe request.
- Streaming, WebSocket, non-replayable, and ambiguous data-plane 404 requests must not be retried.
- Cache hits, misses, evictions, and refetches should be observable, and concurrent misses for one sandbox should be coalesced.

## Proposed approach

Add a bounded process-local cache in the gateway. Query the configured scheduler on cache misses, and populate or refresh entries after successful `LookupNode` calls and successful create or fork assignment recording.

## Compatibility and operational impact

API/config changes:

No client-facing API change is required.

Snapshot or storage format changes:

None.

New host/runtime requirements:

None. The cache is in process memory.

## Alternatives considered

- Keep doing a scheduler lookup for every request: correct but adds avoidable scheduler traffic and latency.
- Use a distributed cache or pub/sub invalidation for every gateway: provides faster invalidation but adds infrastructure and another consistency path.
- Evict only the gateway that handled delete: insufficient because other gateways retain stale mappings.

## Pre-submission checklist

- [x] I searched existing issues and discussions and did not find a duplicate.
- [x] I described a concrete problem and use case, not only a proposed technology.
- [x] I understand that acceptance of the problem does not imply approval of a particular implementation.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the gateway request path that calls the scheduler LookupNode RPC, then inspect ReportSandboxEvent handling and the shared binding store behavior. Done means valid positive routes are cached and coalesced, required evictions and safe refetches work across the stated request types, and cache activity is observable without negative caching.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.