NVIDIA / NVIDIA/OpenShell

feat(audit): scan existing resources at startup and after upgrades with durable operator findings

Open
#3,395 0 comments 0 reactions 1 assignee View on GitHub

@purp is already working on this.

Since Sep 16, 2026.

Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

User Story

As a Platform Admin, I want OpenShell to check existing resources against the running version's operational and security requirements at startup, including after an upgrade, and promptly surface actionable findings so that I can review affected workloads and choose a remedy.

Problem Statement

OpenShell already performs database migrations, selected startup validation, and sandbox lifecycle reconciliation. It lacks a unified operator-facing result for checks of existing resources: which checks completed, which resources need attention, what could not be checked, and which findings remain unresolved after a gateway restart.

Impact / Why This Matters

Operators must combine release notes, manual inspection, and process logs to assess existing workloads after an upgrade. Transient events can be missed, repeated restarts can repeat the same warning, and a successfully started gateway does not tell the operator whether all existing resources passed the relevant checks.

Proposed Design

  • Run applicable checks on gateway startup, including the first startup after an upgrade. Cover persisted sandboxes and workload templates, and compare live runtime state where a check requires it and the driver supports that inspection.
  • Provide a durable findings view for Platform Admins. Each finding identifies its check and revision, affected workspace and resource, severity, evidence summary, first/last observation, resolution state, and recommended action. Keep sensitive values out of notifications.
  • Distinguish a detected problem, missing approval or evidence, and an incomplete check. An incomplete scan must never appear as a clean result; a configuration finding must not be presented as proof of exploitation.
  • Emit a structured event promptly when a finding first appears or materially changes, plus a startup scan summary. Repeated scans and multiple gateway replicas must not produce duplicate unresolved findings or notification storms.
  • Preserve running workloads for operator review. When a check makes further activation conditional on remediation or operator approval, hold affected start/resume operations, including automatic recovery, while keeping administrative access available. Let the Platform Admin inspect, approve where supported, stop, or delete affected workloads explicitly. Acknowledging a notification alone must not satisfy a required approval.
  • Allow Platform Admins to inspect findings and rerun checks through a documented API and CLI workflow. Reuse the existing event and observability work for delivery; client UI integration can follow.

This feature owns scan results, durable finding state, and their operator workflow. Coordinate event production with #2911 and use the delivery surfaces tracked in #1933 and #2762; a new email or webhook service is not required by this proposal. Document the available notification destination and any operator setup required for prompt delivery.

Acceptance Criteria

  • Startup and post-upgrade checks report completion, check revisions, resource coverage, findings, and any incomplete checks.
  • An initial documented check exercises the full scan-to-finding-to-resolution workflow; the feature is not only an empty framework.
  • Findings survive gateway restarts and are discoverable by Platform Admins without subscribing before startup.
  • Unchanged results are deduplicated across repeated scans and gateway replicas; new, materially changed, and resolved findings emit documented events.
  • A failed or unavailable check is visible and cannot be interpreted as a pass.
  • Applicable activation checks run before automatic recovery or explicit start/resume can activate an affected workload.
  • Detection alone does not terminate running workloads; administrative inspection and remediation remain available.
  • Findings distinguish missing approval/evidence from demonstrated unauthorized behavior; notification acknowledgement does not grant approval.
  • Cross-workspace findings are restricted to Platform Admins; any workspace-facing detail obeys workspace authorization and redaction rules.
  • Operator documentation explains inspection, rerun, resolution, notification setup, activation behavior, and coverage limitations.
  • Tests cover restart persistence, repeated scans, multiple replicas, incomplete scans, lifecycle enforcement, notification transitions, and authorization boundaries.

Alternatives Considered

  • Manual post-upgrade commands alone: useful as a rerun option, but easy to omit during deployment.
  • Startup log warnings alone: inexpensive, but provide no durable unresolved-findings view or reliable resolution workflow.
  • Refuse all gateway startup on any finding: prevents administrators from using the API to investigate and remediate unaffected or already-running resources.
  • Automatically terminate every flagged workload: conflates a finding with a confirmed incident and removes the operator's decision about disruption.

Agent Investigation

At source revision 292559c41c1b010de499f1d082583c1aec7b3a4a:

  • crates/openshell-server/src/lib.rs already sequences startup validation and persisted sandbox recovery.
  • crates/openshell-server/src/compute/mod.rs enumerates persisted sandboxes and reconciles driver state. Audit ordering must account for interrupted lifecycle recovery as well as the ordinary startup sweep.
  • Existing sandbox watch and event buses provide live delivery, but their event buffers are in memory. Sandbox conditions are also rebuilt from driver observations, so durable findings need explicit preservation and ownership.
  • The generic persistence store and structured event builders offer reusable foundations. Existing planned event export and subscription work should supply delivery rather than be duplicated here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.