kvcache-ai / kvcache-ai/AgentENV
Warm Firecracker pool never decays its refill target after a burst
- Dominant language
- Rust
- Stars
- 3.5k
- Forks
- 309
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 66
Description
**Affected component:** Firecracker runtime
## Problem statement
As an operator running AgentENV with the warm Firecracker process pool enabled,
I need the pool's refill target to decay after the burst that raised it has passed,
because the current behavior pins the excess warm processes — and the memory they
hold — for the entire lifetime of the server process.
## Use case
Sandbox hosts serving bursty workloads (e.g. CI/agent sandbox creation in business
hours, idle overnight). A single burst of sandbox creations ratchets the geometric
fill target up; hours later the host still holds that many parked Firecracker
processes. On a shared host this is pure memory waste and also keeps more processes
parked with `oom_score_adj=1000` than the steady state needs.
## Current behavior and workarounds
`WarmPool`'s maintenance worker computes its fill target from a geometric
high-watermark that only ever ratchets up on acquisitions. There is no time-based
decay: once raised, the target stays for the process lifetime. The only workaround
is restarting the server, which defeats the purpose of a warm pool.
## Desired behavior
- Given a configurable idle TTL (seconds),
- When no warm-process acquisition happens within the TTL,
- Then the maintenance worker decays the refill target back to the configured low
watermark and drains the excess warm processes.
- Any acquisition resets the idle clock (busy pools never decay).
- `0` disables decay and preserves the current ratchet behavior.
## Proposed approach
Add `PoolConfig::idle_ttl` to the `warm-pool` crate: the maintenance loop tracks
the last-acquisition instant, wakes on the TTL boundary, and when expired reduces
the fill target toward the low watermark and drains the excess entries. Wire it as
`[pool.firecracker] idle_ttl_secs` (default 600, `0` disables), documented in
`config/default.toml` and the configuration reference. The network-slot and ublk
pools keep `idle_ttl: None` (historical ratchet): they hold no guest processes, so
idle decay buys nothing there.
Implementation ready: https://github.com/ResultadosDigitais/AgentENV (branch
`upstream/warm-pool-idle-ttl`), with unit tests for decay, TTL wake-up, and
clock reset.
## Compatibility and operational impact
API/config changes: new optional key `[pool.firecracker] idle_ttl_secs` (default
600). No breaking change; setting `0` restores exact previous behavior.
Snapshot or storage format changes: none.
New host/runtime requirements: none.
Upgrade and rollback considerations: rolling back simply ignores the new key;
behavior with the key absent on old builds is unchanged.
## Alternatives considered
- Time-based decay of *parked entries themselves* (per-entry TTL): wastes warm
processes that a steady-state workload still needs; decaying the refill target
preserves the configured low watermark.
- External memory-pressure-driven scaling: couples pool sizing to host-level
signals the pool does not currently consume; a TTL is deterministic and
testable.
- Doing nothing (restart to reset): operationally hostile for long-lived hosts.
## Non-goals
- Changing the geometric growth policy itself.
- Per-entry idle TTL for parked processes.
- Applying decay to the network-slot or ublk pools.
## Contribution
I can implement it (implementation ready, PR to follow).
## Pre-submission checklist
- [x] I searched existing issues and discussions and did not find a duplicate.
- [x] I described a concrete problem and use case, not only a proposed technology.
- [x] I understand that acceptance of the problem does not imply approval of a particular implementation.
Contributor guide
Research direction
Start in the warm-pool crate with PoolConfig and the maintenance worker that computes the geometric fill target. Review the existing unit tests and configuration entries in config/default.toml and the configuration reference. Done means Firecracker decay honors the idle TTL and clock resets, 0 preserves ratcheting, excess processes drain, and network-slot and ublk pools remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100