rivet-dev / rivet-dev/agentos

Guest-execution slot counter has no near-threshold warning

Open
#1,912 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
4.6k
Forks
251
Avg merge
2d 15h
Merged PRs (30d)
34

Description

Problem

The process-wide guest-execution admission counter goes from silent to rejecting with nothing in between. Every other bounded limit in the repo warns first.

CLAUDE.md requires that every limit "be bounded by default, warn near threshold, and fail with a typed error that names the limit and how to raise it." This counter satisfies the first and third, not the second.

Where

SlotControl is a bare Arc<(Mutex<usize>, Condvar)> (crates/v8-runtime/src/session.rs:1047). Admission is fail-fast:

// crates/v8-runtime/src/session.rs — SessionSlotPermit::try_acquire
if *active >= maximum {
    return Err(format!("ERR_AGENTOS_GUEST_EXECUTION_LIMIT: ..."));
}
*active += 1;
metrics.observe_executor(ExecutorMetricClass::Vm, *active, 0);

RuntimeMetrics::observe_executor (crates/runtime/src/metrics.rs:477) only stores gauge values — it has no threshold evaluation and emits no warning:

pub fn observe_executor(&self, class: ExecutorMetricClass, active: usize, queued: usize) {
    executor.active.observe(active);
    executor.queued.observe(queued);
}

Meanwhile agentos_bridge::queue_tracker gives every registered limit edge-triggered warnings at WARN_FILL_PERCENT = 80 with hysteresis re-arm at REARM_FILL_PERCENT = 50. The slot counter is not registered with it.

Why it matters

This is the limit a parallel-agent fleet actually hits, and it's held for the whole lifetime of every guest process (JS, TS, Python, WASM command alike) — not just while it burns CPU. A host sitting at 63/64 looks identical in the logs to one sitting at 3/64, right up until an execution is rejected outright. There is no queue and no retry, so the first sign of trouble is a failed agent.

An 80% warning would have turned a run that dies at peak concurrency into a visible "you are near the ceiling, raise it" signal beforehand.

Suggested fix

Small, and the pieces already exist:

  • crates/v8-runtime already depends on agentos-bridge (crates/v8-runtime/Cargo.toml:13).
  • LimitCategory::Resource is documented as exactly this shape — "a saturating resource counter (fds, processes, sockets, bytes in use)".

Add a TrackedLimit::ActiveGuestExecutions variant, register it in SessionManager::new via register_limit(..., max_concurrency), and call QueueGauge::observe_depth(*active) from both SessionSlotPermit::try_acquire and its Drop. That yields the standard 80% warning, plus depth/high-water/capacity in queue_snapshot for free.

Worth confirming the warning is reachable at the default ceiling of 64 without a test that has to saturate real V8 isolates — the counter can be exercised directly, as the existing permit tests do.

Related

Adjacent to #1816, but not covered by it: that issue is scoped to VmLimits fields and per-VM observability, and its proposed audit ("the limits audit fails when a measurable limit lacks accounting") is keyed on VmLimits. This ceiling is process-scoped and deliberately not a VmLimits field, so it would fall through that net.

Context: found while making the ceiling operator-tunable on branch claude/maxactivevms-exposure-analysis-nd6qqa, which added AGENTOS_MAX_ACTIVE_GUEST_EXECUTIONS and a startup log of the effective value. The startup line tells you the ceiling; nothing tells you how close you are to it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in crates/v8-runtime/src/session.rs at SlotControl and SessionSlotPermit::try_acquire, then read RuntimeMetrics::observe_executor in crates/runtime/src/metrics.rs and agentos_bridge::queue_tracker. Run the existing permit tests and exercise the counter directly without saturating V8 isolates. Done means near-threshold warnings and queue depth/high-water/capacity visibility appear on acquisition and release at the default ceiling.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, wasm
Domain
backend, observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.