NVIDIA / NVIDIA/OpenShell

feat(cli): add sandbox prune command to delete ERROR-phase sandboxes

Open
#2,594 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

state:triage-needed
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

Problem Statement

When sandboxes fail during provisioning or runtime, they remain in the ERROR phase indefinitely. Users currently have two options:

  • delete them individually by name with sandbox delete <name>, or
  • wipe everything with sandbox delete --all

There is no way to clean up only the broken sandboxes while leaving healthy ones running.

This is especially painful in environments where multiple sandboxes are running concurrently - a single failed experiment shouldn't require manually identifying and deleting each errored sandbox, and --all is too destructive.

Proposed Design

Add openshell sandbox prune - a CLI command backed by a new PruneSandboxes gRPC RPC that performs server-side filtering and deletion. The semantics are similar to podman container prune and docker container prune.

Proto:

  • New PruneSandboxes RPC on the OpenShell service with sandbox:write scope and user workspace role (matching DeleteSandbox).
  • PruneSandboxesRequest takes workspace and all_workspaces (admin-only, mirrors ListSandboxes).
  • PruneSandboxesResponse returns repeated string pruned_names and repeated string failed_names for best-effort semantics - one sandbox failing to delete doesn't abort the rest.
Alternatives Considered
  1. Client-side filtering - CLI lists all sandboxes, filters for ERROR, then issues individual DeleteSandbox RPCs. Simpler (no proto changes), but introduces a TOCTOU race between list and delete, produces N round trips, and pushes logic into every client (CLI, SDK, Python SDK).
  2. Add a phase filter to ListSandboxes. More general, but still requires client-side delete loop. The prune operation is common enough to warrant a dedicated RPC that does both steps atomically.
  3. Extend DeleteSandbox with a phase filter. Overloads the existing RPC's semantics. A separate RPC is cleaner and doesn't risk breaking existing callers.
Agent Investigation
  • Explored the full DeleteSandbox call chain: CLI (run::sandbox_delete) → gRPC
    → gateway handler (handle_delete_sandbox) → ComputeRuntime::delete_sandbox()
    → compute driver. The handler is a thin authz wrapper; all complexity lives in
    the compute layer which is safe to reuse.
  • Confirmed SandboxPhase enum has ERROR = 3 and the established pattern for
    checking phase is SandboxPhase::try_from(sandbox.phase()).ok() ==
    Some(SandboxPhase::Error).
  • Confirmed the gateway's existing reconciliation loop
    (reconcile_store_with_backend) already prunes orphan sandboxes on a timer, but
    does not prune by phase — this is a separate concern.
  • Confirmed ListSandboxesRequest has no phase filter field, so client-side
    filtering would require fetching all sandboxes.
Checklist
  • I've reviewed existing issues and the architecture docs
  • This is a design proposal, not a "please build this" request

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing CLI run::sandbox_delete and trace the mentioned gateway handler handle_delete_sandbox through ComputeRuntime::delete_sandbox and the compute driver. Review the service proto and ListSandboxesRequest/DeleteSandbox authorization patterns. Done means a server-side PruneSandboxes RPC and openshell sandbox prune command support ERROR-phase filtering, best-effort results, and the stated workspace permissions.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, rust
Domain
api, backend, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.