Guest-execution slot counter has no near-threshold warning
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 4.6k
- Forks
- 251
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 34
Description
Problem
The process-wide guest-execution admission counter goes from silent to rejecting with nothing in between. Every other bounded limit in the repo warns first.
CLAUDE.md requires that every limit "be bounded by default, warn near threshold, and fail with a typed error that names the limit and how to raise it." This counter satisfies the first and third, not the second.
Where
SlotControl is a bare Arc<(Mutex<usize>, Condvar)> (crates/v8-runtime/src/session.rs:1047). Admission is fail-fast:
// crates/v8-runtime/src/session.rs — SessionSlotPermit::try_acquire
if *active >= maximum {
return Err(format!("ERR_AGENTOS_GUEST_EXECUTION_LIMIT: ..."));
}
*active += 1;
metrics.observe_executor(ExecutorMetricClass::Vm, *active, 0);
RuntimeMetrics::observe_executor (crates/runtime/src/metrics.rs:477) only stores gauge values — it has no threshold evaluation and emits no warning:
pub fn observe_executor(&self, class: ExecutorMetricClass, active: usize, queued: usize) {
executor.active.observe(active);
executor.queued.observe(queued);
}
Meanwhile agentos_bridge::queue_tracker gives every registered limit edge-triggered warnings at WARN_FILL_PERCENT = 80 with hysteresis re-arm at REARM_FILL_PERCENT = 50. The slot counter is not registered with it.
Why it matters
This is the limit a parallel-agent fleet actually hits, and it's held for the whole lifetime of every guest process (JS, TS, Python, WASM command alike) — not just while it burns CPU. A host sitting at 63/64 looks identical in the logs to one sitting at 3/64, right up until an execution is rejected outright. There is no queue and no retry, so the first sign of trouble is a failed agent.
An 80% warning would have turned a run that dies at peak concurrency into a visible "you are near the ceiling, raise it" signal beforehand.
Suggested fix
Small, and the pieces already exist:
crates/v8-runtimealready depends onagentos-bridge(crates/v8-runtime/Cargo.toml:13).LimitCategory::Resourceis documented as exactly this shape — "a saturating resource counter (fds, processes, sockets, bytes in use)".
Add a TrackedLimit::ActiveGuestExecutions variant, register it in SessionManager::new via register_limit(..., max_concurrency), and call QueueGauge::observe_depth(*active) from both SessionSlotPermit::try_acquire and its Drop. That yields the standard 80% warning, plus depth/high-water/capacity in queue_snapshot for free.
Worth confirming the warning is reachable at the default ceiling of 64 without a test that has to saturate real V8 isolates — the counter can be exercised directly, as the existing permit tests do.
Related
Adjacent to #1816, but not covered by it: that issue is scoped to VmLimits fields and per-VM observability, and its proposed audit ("the limits audit fails when a measurable limit lacks accounting") is keyed on VmLimits. This ceiling is process-scoped and deliberately not a VmLimits field, so it would fall through that net.
Context: found while making the ceiling operator-tunable on branch claude/maxactivevms-exposure-analysis-nd6qqa, which added AGENTOS_MAX_ACTIVE_GUEST_EXECUTIONS and a startup log of the effective value. The startup line tells you the ceiling; nothing tells you how close you are to it.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in crates/v8-runtime/src/session.rs at SlotControl and SessionSlotPermit::try_acquire, then read RuntimeMetrics::observe_executor in crates/runtime/src/metrics.rs and agentos_bridge::queue_tracker. Run the existing permit tests and exercise the counter directly without saturating V8 isolates. Done means near-threshold warnings and queue depth/high-water/capacity visibility appear on acquisition and release at the default ceiling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, wasm
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100