NVIDIA / NVIDIA/OpenShell

feat: fail-closed capability negotiation for gateway↔supervisor signals under version skew

Open
#2,949 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

state:triage-needed
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

References: originates from PR #2852 (--no-login-shell, which Closes #2668) and review finding GATOR-7d79a247-01 on that PR.

Related issues: #1731 (push gateway-owned desired state — deliberately excludes capability negotiation for the config-push pipeline; see Relationship below), #2589 (in-place supervisor upgrade via re-exec — complementary root-cause fix; see Alternatives).

Suggested labels: area:gateway, area:supervisor, topic:security (this gates a security-relevant opt-out; not a vulnerability disclosure — no live exploit — so it is filed as a feature per SECURITY.md).

User Story

As an operator running long-lived sandboxes who upgrades the gateway in place, I want the gateway to reject a requested sandbox behavior when the sandbox's supervisor doesn't support it, so that a security- or correctness-relevant option never silently degrades to the old behavior without any signal.

Problem Statement

The supervisor is baked into each sandbox at creation and never updates for that sandbox's lifetime, while the gateway is upgraded independently. When a newer gateway sends a behavior signal to an older supervisor that predates it, the supervisor accepts the request but ignores the signal — there is no mechanism for the gateway to learn which behaviors the live supervisor actually understands. Today these signals are carried as SSH environment requests, which an older supervisor acknowledges with success even when it does not act on them.

Impact / Why This Matters

Without this, a gateway-issued option can silently no-op against an already-running sandbox, and the caller receives a normal success as if it had taken effect.

  • Concrete case: --no-login-shell (PR #2852, addressing #2668). Against a pre-change supervisor, the command still runs through a login shell (bash -lc), the user's profile files are sourced, and the caller sees a clean success. For a security opt-out on a trusted exec boundary, silent failure is worse than an explicit error.
  • This is not exec-only. There are already two such signals riding SSH env requests — no_login_shell (exec) and main_read_only (main attachment) — so the skew spans more than one pipeline (exec, interactive shell/PTY, direct-tcpip tunnels, subsystems, main attach all share the same gateway↔supervisor SSH connection).
  • The skew can occur within a single normal deploy (a sandbox created just before a gateway upgrade), not only across releases, so a pre-1.0 "we break things" stance does not remove the exposure.

Current workaround: none. The gateway cannot tell an honored signal from a silently dropped one.

Proposed Design

From the operator's and caller's perspective:

  • When a caller requests a behavior the live supervisor supports, it works as today.
  • When a caller requests a behavior the live supervisor does not support, the operation fails fast with a clear, actionable error (e.g. gRPC FailedPrecondition: "supervisor predates this capability; recreate the sandbox") instead of silently running the old behavior.
  • The capability check is negotiated once per SSH session and applies to every pipeline on that connection (exec, interactive, tunnels, subsystems, main attach), not re-solved per feature.
  • Adding a new skew-sensitive behavior in the future should not require re-inventing detection — it should reuse the same negotiation and the same fail-closed path.

Internal wire format is left to implementers; directions are in Alternatives.

Relationship to #1731 (why this is not the rejected negotiation)

#1731 ("Push gateway-owned desired state to supervisors") resolved to not add capability negotiation, a protocol-version field, or a rollout mode, using a required bootstrap exchange plus bounded timeout as its compatibility gate. That decision is scoped to the config/policy/provider/inference push pipeline over the reverse ConnectSupervisor session, where the gateway controls both ends of a single required exchange and can fail the whole init if the bootstrap does not complete.

This issue is a different pipeline: per-request behavior signals carried as SSH env requests (exec, interactive/PTY, tunnels, subsystems, main attach). Two properties make #1731's gate insufficient here:

  • These signals are optional and per-operation, not a single required init exchange. An older supervisor accepts the SSH session, the env request, and the channel — and replies channel_success unconditionally (crates/openshell-supervisor-process/src/ssh.rs:726) — so there is no bootstrap-style all-or-nothing gate to hang the result on.
  • The failure is silent per-command, not at init: the session is healthy, only the individual behavior no-ops.

So the fail-closed check must be consulted at each affected call site, keyed on something learned once per session. This does not reintroduce config-push negotiation; it reuses #1731's spirit (learn compatibility once, fail clearly) on the SSH transport. If maintainers prefer to fold this into #1731's bootstrap result (advertise SSH-behavior capabilities in SessionAccepted rather than a new SupervisorHello), that satisfies this issue's acceptance criteria — see Alternative 4.

Acceptance Criteria

  • A newer gateway paired with an older supervisor rejects an unsupported behavior with a clear FailedPrecondition-style error rather than silently falling back.
  • The reproduction for no_login_shell fails closed: with a profile marker seeded, requesting a non-login exec against a pre-change supervisor returns an unsupported-feature error and does not source the profile.
  • Capability support is determined once per session and consulted by all affected pipelines (exec unary + interactive at minimum).
  • When the supervisor supports the capability, behavior is unchanged (no regression, no added per-command round-trip).
  • Adding a future skew-sensitive signal reuses the negotiation without a new detection mechanism.

Alternatives Considered

  1. SupervisorHello capability set (preferred direction). Supervisor advertises a set of supported capability strings once per session; gateway caches it on the live session and gates each behavior against it. Additive (new behavior = new string), composes across pipelines, no version-ordering tables. Cost: a new negotiation message/contract to define and maintain, and one exchange at session open.
  2. SSH identification banner version gate. Supervisor stamps its OpenShell version into the SSH identification string (SSH-2.0-<software>), already exchanged at connection start; gateway parses the remote id and gates on a version threshold. No new message, no proto field, no extra round-trip. Narrower: carries a version, not a capability list, so each future behavior needs its own "added in version ≥ X" mapping on the gateway side. (Verified feasible in russh 0.62: supervisor can set server::Config.server_id; client exposes Session::remote_sshid() — though it lives on the client Session, not the Handle, so the gateway must capture it in a Handler callback and thread it out to the exec path.)
  3. Do nothing (rely on pre-1.0 breakage tolerance). Rejected: the exposure occurs in normal in-place upgrades, and the failure mode is a silent no-op on a security-relevant option.
  4. Fold capabilities into #1731's SessionAccepted bootstrap. Rather than a new SupervisorHello, the supervisor advertises its SSH-behavior capability set as one field of the existing bootstrap result #1731 already requires. The gateway caches it on the live session and gates behaviors identically. Reuses an exchange that must exist anyway (no new message, no extra round-trip) and stays consistent with #1731's "learn compatibility once" gate. Cost: couples this fail-closed path to #1731 landing first, and mixes SSH-transport concerns into a config-push message. Viable if #1731 ships before or with this work.
  5. In-place supervisor upgrade via re-exec (#2589). If the supervisor can replace its own binary in place, a live sandbox's supervisor gains new capabilities without recreation and the skew disappears at the source. Complementary, not competing: re-exec requires release-matched artifacts and does not help a sandbox whose supervisor has not yet been re-exec'd, so a fail-closed negotiation is still needed for the window before/without upgrade. Out of scope here; tracked in #2589.

Direction (1)/(4) is preferred because the skew already spans multiple pipelines and is expected to grow; a session-scoped capability set turns "re-solve skew per feature" into "add one string," at zero steady-state cost. Choice between a standalone SupervisorHello (1) and folding into #1731's bootstrap (4) depends on sequencing of #1731.

Agent Investigation

  • Gateway emits the signal at two exec sites: crates/openshell-server/src/grpc/sandbox.rs:2170 (interactive) and :2359 (unary) — both via set_env(false, NO_LOGIN_SHELL_ENV.0, ...) (want_reply=false). These are the sites gator flagged.
  • Supervisor consumes it in crates/openshell-supervisor-process/src/ssh.rs: env_request (fn at :701) — no_login_shell variable check at :715/set at :718, main_read_only at :720; the flag maps to the shell argument in login_shell_flag at :1058 (-c vs -lc), covered by test login_shell_flag_controls_profile_sourcing at :1778.
  • env_request unconditionally replies channel_success at :726 for any variable, so an older supervisor acks an unknown signal with success — there is no reply-based signal to detect a dropped capability (this rules out flipping want_reply on the env request).
  • Two signals already ride SSH env requests today: OPENSHELL_NO_LOGIN_SHELL (exec) and OPENSHELL_MAIN_READ_ONLY (main attachment), confirming the pattern spans pipelines.
  • russh 0.62 supports both halves of the banner approach: server::Config.server_id (set) and Session::remote_sshid() (read, client side), with the plumbing caveat noted above.
  • Duplicate scan (issues, all states): no existing issue proposes capability negotiation. Closest are #1731 (config-push, explicitly excludes negotiation — see Relationship) and #2589 (re-exec upgrade, complementary — see Alternatives). Parent bug #2668 is state:accepted; origin PR #2852 is still open.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the gateway sites in crates/openshell-server/src/grpc/sandbox.rs:2170 and :2359, then read env_request in crates/openshell-supervisor-process/src/ssh.rs:701 and the login_shell_flag test at :1778. Trace how one SSH session is established and how capabilities could cover both exec paths and main attachment. Done means unsupported signals fail with a clear precondition error, supported behavior is unchanged, and the no_login_shell profile-sourcing reproduction is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, distributed-systems, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.