bug: gator supervisor retries transient failures forever despite a documented cap

Aperta
#3,322 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

@purp ci sta già lavorando.

Dal 14/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

state:triage-needed

User Story

As a maintainer supervising a gator agent, I want the supervisor to stop after repeated transient failures, so that a wedged or permanently-failing gator surfaces for human attention instead of burning sandbox time indefinitely.

Problem Statement

OPENSHELL_AGENT_MAX_TRANSIENT_FAILURES (default 5) is documented as a retry cap, but it does not cap anything. In scripts/agents/runtime/supervisor.sh, retry_watch_cycle() uses the value only to choose between two nearly identical log messages via a modulo test; both branches fall through to sleep_with_heartbeat and return, and the watch loop continues unconditionally. There is no exit path on repeated transient failures.

The operator-facing documentation states the opposite. .agents/skills/launch-openshell-gator/SKILL.md:313 tells the operator to "Let supervisor retry unless repeated failures hit the configured cap," which describes behavior that does not exist.

Impact / Why This Matters

A gator hitting a persistent auth, transport, or infrastructure failure is classified as transient_failure and retries forever. Because the supervisor never exits, the sandbox stays alive and no terminal_failure sentinel is ever written, so the documented operator signal for "this needs a human" never fires.

The current workaround is for an operator to notice the condition by reading logs and delete the sandbox manually. That is insufficient because the sentinel table is the documented mechanism for deciding whether human action is needed, and it cannot report a state the supervisor cannot reach. Nothing else bounds the run: there is no timeout or cleanup path elsewhere in the gator lifecycle, so an unattended failure is only ever found by someone going to look.

Acceptance Criteria

  • With OPENSHELL_AGENT_MAX_TRANSIENT_FAILURES=N, the supervisor stops retrying after N consecutive transient failures rather than continuing indefinitely.
  • On hitting the cap, the supervisor emits a terminal_failure sentinel with a reason identifying the transient failure, matching the sentinel contract the skill documents.
  • Consecutive-failure counting resets after a successful cycle, so intermittent failures spread over a long run do not accumulate to the cap.
  • OPENSHELL_AGENT_MAX_TRANSIENT_FAILURES=0 retains its existing meaning of unlimited retries, and that meaning is documented.
  • .agents/skills/launch-openshell-gator/SKILL.md describes the actual behavior, including what the operator should do when the cap is hit.

Reproduction Steps

  1. Launch a gator with a low cap: OPENSHELL_AGENT_MAX_TRANSIENT_FAILURES=2 ./scripts/agents/run.sh --agent gator ...
  2. Induce a repeatable transient failure, for example by invalidating the gh credential the sandbox uses so watch cycles fail on transport/auth.
  3. Watch the supervisor log.

Observed: openshell-agent: transient watch failure N (<reason>); retrying in Ns repeats without bound; the wording alternates at multiples of the cap but the loop never exits and no terminal_failure sentinel is written.

Expected: the supervisor stops after 2 consecutive transient failures and reports status=terminal_failure.

Environment

  • OpenShell: repository scripts at 5b9daab93 (not a released artifact)
  • OS: macOS 15 (Darwin 25.6.0); the supervisor is plain bash and the defect is host-independent
  • Component: scripts/agents/runtime/supervisor.sh, gator agent runtime

Logs

scripts/agents/runtime/supervisor.sh:21   MAX_TRANSIENT_FAILURES="${OPENSHELL_AGENT_MAX_TRANSIENT_FAILURES:-5}"
scripts/agents/runtime/supervisor.sh:140      if [[ "$MAX_TRANSIENT_FAILURES" -gt 0 ]]; then
scripts/agents/runtime/supervisor.sh:141          if [[ $((transient_failures % MAX_TRANSIENT_FAILURES)) -eq 0 ]]; then
                                                  # both branches only echo, then sleep and return
Lingua principale
Rust
Stelle
8.7k
Fork
1.3k
Merge medio
2g 7h
PR unite (30g)
243

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/OpenShell

Tutte le issue di NVIDIA/OpenShell

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.