NVIDIA / NVIDIA/OpenShell

bug: gator supervisor retries transient failures forever despite a documented cap

Open
#3,322 0 comments 0 reactions 1 assignee View on GitHub

@purp is already working on this.

Since Sep 14, 2026.

state:triage-needed
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

User Story

As a maintainer supervising a gator agent, I want the supervisor to stop after repeated transient failures, so that a wedged or permanently-failing gator surfaces for human attention instead of burning sandbox time indefinitely.

Problem Statement

OPENSHELL_AGENT_MAX_TRANSIENT_FAILURES (default 5) is documented as a retry cap, but it does not cap anything. In scripts/agents/runtime/supervisor.sh, retry_watch_cycle() uses the value only to choose between two nearly identical log messages via a modulo test; both branches fall through to sleep_with_heartbeat and return, and the watch loop continues unconditionally. There is no exit path on repeated transient failures.

The operator-facing documentation states the opposite. .agents/skills/launch-openshell-gator/SKILL.md:313 tells the operator to "Let supervisor retry unless repeated failures hit the configured cap," which describes behavior that does not exist.

Impact / Why This Matters

A gator hitting a persistent auth, transport, or infrastructure failure is classified as transient_failure and retries forever. Because the supervisor never exits, the sandbox stays alive and no terminal_failure sentinel is ever written, so the documented operator signal for "this needs a human" never fires.

The current workaround is for an operator to notice the condition by reading logs and delete the sandbox manually. That is insufficient because the sentinel table is the documented mechanism for deciding whether human action is needed, and it cannot report a state the supervisor cannot reach. Nothing else bounds the run: there is no timeout or cleanup path elsewhere in the gator lifecycle, so an unattended failure is only ever found by someone going to look.

Acceptance Criteria

  • With OPENSHELL_AGENT_MAX_TRANSIENT_FAILURES=N, the supervisor stops retrying after N consecutive transient failures rather than continuing indefinitely.
  • On hitting the cap, the supervisor emits a terminal_failure sentinel with a reason identifying the transient failure, matching the sentinel contract the skill documents.
  • Consecutive-failure counting resets after a successful cycle, so intermittent failures spread over a long run do not accumulate to the cap.
  • OPENSHELL_AGENT_MAX_TRANSIENT_FAILURES=0 retains its existing meaning of unlimited retries, and that meaning is documented.
  • .agents/skills/launch-openshell-gator/SKILL.md describes the actual behavior, including what the operator should do when the cap is hit.

Reproduction Steps

  1. Launch a gator with a low cap: OPENSHELL_AGENT_MAX_TRANSIENT_FAILURES=2 ./scripts/agents/run.sh --agent gator ...
  2. Induce a repeatable transient failure, for example by invalidating the gh credential the sandbox uses so watch cycles fail on transport/auth.
  3. Watch the supervisor log.

Observed: openshell-agent: transient watch failure N (<reason>); retrying in Ns repeats without bound; the wording alternates at multiples of the cap but the loop never exits and no terminal_failure sentinel is written.

Expected: the supervisor stops after 2 consecutive transient failures and reports status=terminal_failure.

Environment

  • OpenShell: repository scripts at 5b9daab93 (not a released artifact)
  • OS: macOS 15 (Darwin 25.6.0); the supervisor is plain bash and the defect is host-independent
  • Component: scripts/agents/runtime/supervisor.sh, gator agent runtime

Logs

scripts/agents/runtime/supervisor.sh:21   MAX_TRANSIENT_FAILURES="${OPENSHELL_AGENT_MAX_TRANSIENT_FAILURES:-5}"
scripts/agents/runtime/supervisor.sh:140      if [[ "$MAX_TRANSIENT_FAILURES" -gt 0 ]]; then
scripts/agents/runtime/supervisor.sh:141          if [[ $((transient_failures % MAX_TRANSIENT_FAILURES)) -eq 0 ]]; then
                                                  # both branches only echo, then sleep and return

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.