NVIDIA / NVIDIA/OpenShell

bug: distinguish intentional signal stops from runtime restarts

Open
#3,083 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:compute area:sandbox state:stale
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

User Story

As an OpenShell operator, I want sandbox status to distinguish an intentional shutdown from a runtime interruption, so that stopped sandboxes are not presented as having restarted unexpectedly and real runtime restarts remain recoverable.

Problem Statement

The Docker and Podman drivers currently classify exits 137 (SIGKILL) and 143 (SIGTERM) as ContainerRuntimeRestart. Those codes establish only that a process was terminated by a signal; they do not identify the sender or intent. An explicit gateway stop that forwards SIGTERM therefore produces the same condition as a Podman/Docker machine or daemon restart.

The durable Stopping phase now prevents that ambiguity from promoting an in-flight explicit stop to Error, but a delayed watcher snapshot can still arrive after Stopped is persisted and replace the user-visible status reason with ContainerRuntimeRestart.

Impact / Why This Matters

Operators can see a sandbox in Stopped phase with a contradictory runtime-restart condition after a normal stop. More broadly, treating all 137/143 exits as runtime restarts conflates graceful stop, forced timeout kill, external intervention, and genuine runtime interruption. The current workaround is to infer intent from lifecycle phase, which protects the immediate flow but does not make the driver status semantically precise.

Acceptance Criteria

  • An explicit gateway stop remains Stopped when a late Docker or Podman signal-exit snapshot arrives, and its terminal status continues to report the intentional stop.
  • A signal termination without explicit stop intent remains distinguishable from a confirmed runtime interruption.
  • Gateway restart recovery continues to recover sandboxes interrupted by a real Docker or Podman runtime/machine restart.
  • OOM termination and ordinary application exits keep their existing distinct behavior.
  • Regression coverage covers Docker and Podman for explicit SIGTERM stop, forced SIGKILL timeout, delayed watcher delivery, OOM, and runtime/machine restart.

Reproduction Steps

  1. Start a Docker- or Podman-backed sandbox.
  2. Stop it through the gateway so the supervisor forwards SIGTERM to its workload.
  3. Observe the driver report exit 143 as ContainerRuntimeRestart.
  4. Deliver that watcher snapshot after the gateway has persisted Stopped.
  5. Observe the sandbox phase remain Stopped while its condition reason no longer reflects the intentional stop.

Environment

  • OpenShell: current main development build
  • Compute drivers: Docker and rootless Podman
  • Related issue: #2855
  • Historical recovery behavior: #2179

Agent Investigation

ContainerRuntimeRestart is currently a heuristic for exit 137/143 in both Docker and Podman. The exit status has no provenance, so operation intent and independently observed runtime state must be considered separately.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the Docker and Podman driver handling of exit codes 137/143, then follow how gateway stop intent and delayed watcher snapshots update sandbox status. Review the existing recovery behavior related to issues #2855 and #2179, and run or extend regression coverage for both drivers. Done means intentional stops remain terminal and distinguishable while genuine runtime restarts remain recoverable, with OOM and ordinary exits unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
backend, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.