NVIDIA / NVIDIA/OpenShell

Gateway acknowledges an MCP policy approval before the sandbox confirms it can load the resulting policy

Open
#2,905 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

state:stale
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

User Story

As an OpenShell policy reviewer, I need an approved policy change to be reported as applied only after the sandbox has successfully loaded it, so that a gateway acknowledgement cannot leave the sandbox on a failed or ambiguous policy revision.

Problem Statement

During a long-running agent evaluation, an advisor proposed an MCP egress rule equivalent to:

  • host: mcp.example.com. (note the trailing dot)
  • protocol: mcp
  • port: 443
  • MCP method: tools/call
  • MCP tool/path: get_file_contents
  • enforcement: enforce
  • scoped to a concrete client binary

The candidate policy was materialized, validation reported no new findings, and the reviewer approved it. ApproveDraftChunk returned success and the gateway emitted an approval/configuration event for policy version 43.

Approximately two seconds later, the sandbox reported that version 43 had failed to load. Subsequent policy versions 44 through 46 also failed to load, and the sandbox did not recover to a known-good revision. The agent's next inference request disconnected shortly after the first load failure, and the run eventually stopped.

The trailing dot is the leading suspected trigger: a fully qualified hostname ending in . contains an empty final DNS label, which may be rejected by the runtime host-pattern parser. However, the exact sandbox load_error was not retained in the run artifacts, so this specific validator diagnosis is not proven.

Regardless of the exact parser failure, the observed consistency bug is clear: the gateway acknowledged and persisted the approval before confirming that the sandbox could load the resulting policy, and the failed candidate remained the basis for later policy composition.

Impact / Why This Matters

  • The control plane can report a policy change as successfully applied when the data plane has rejected it.
  • Operators and agents receive a misleading success signal and may continue under false assumptions about the active policy.
  • One malformed rule can poison subsequent policy revisions instead of being isolated or rolled back.
  • In this observation, the failure occurred about 2 hours 30 minutes into a planned 6-hour run; the sandbox stopped about 15 minutes later, invalidating the accumulated work.
  • The initiating rule came from an otherwise normal agent-generated MCP policy suggestion, so this is reachable in automated policy-review workflows.

Reproduction

  1. Start an OpenShell sandbox with policy review and dynamic policy updates enabled.
  2. Submit an MCP allow rule whose host has a trailing dot, for example mcp.example.com..
  3. Include a normal MCP constraint such as method tools/call and tool/path get_file_contents.
  4. Approve the proposal through the normal draft-chunk approval path.
  5. Observe the gateway's approval response and the sandbox's subsequent policy status report.

Observed behavior:

  • The approval RPC succeeds and the gateway reports the new version as approved/applied.
  • The sandbox subsequently reports that the same policy version failed to load.
  • Later candidate versions continue to inherit the invalid state and also fail.

If the trailing-dot rule is normalized or rejected earlier in the current implementation, the lifecycle inconsistency can be reproduced with any candidate that passes gateway-side validation but fails the sandbox's exact runtime loader.

Expected Behavior

  • A policy revision should not be reported as applied until the target sandbox confirms that it loaded successfully.
  • Candidate policy validation should use the same parser and constraints as the sandbox runtime.
  • If a sandbox rejects a revision, OpenShell should preserve or restore the last successfully loaded policy and should not use the failed candidate as the base for subsequent revisions.
  • The sandbox load error should be surfaced clearly to the reviewer/operator.

Proposed Solution

  1. Validate the fully materialized candidate policy using the exact runtime policy loader before accepting the approval.
  2. Normalize hostnames consistently across proposal creation, validation, and runtime loading, or reject trailing-dot host patterns with an actionable validation error.
  3. Model the lifecycle explicitly, for example: gateway_accepted, sandbox_loaded, and sandbox_load_failed. Do not label a revision applied at the first stage.
  4. On sandbox_load_failed, automatically retain or roll back to the last sandbox-confirmed revision.
  5. Exclude failed candidates from the base used for the next composition attempt.
  6. Persist and expose the sandbox's load_error in gateway logs, proposal status, and API responses.

Acceptance Criteria

  • A candidate rejected by the sandbox runtime is never reported as applied.
  • Gateway-side validation and sandbox loading agree for MCP host patterns, including trailing-dot inputs.
  • A trailing-dot hostname is either normalized consistently or rejected before approval with a clear error.
  • A failed policy revision leaves the sandbox operating on the last successfully loaded revision.
  • Subsequent proposals are composed from the last successfully loaded revision, not the failed candidate.
  • The exact sandbox load error is retained and visible to operators and reviewers.
  • An integration test covers approval followed by a sandbox load failure and verifies status, rollback, and subsequent composition behavior.

Environment

  • OpenShell commit: d9d38a561944dd041342c004dbfe6ff5b3c8d8de
  • Linux ARM64 sandbox running under Docker
  • Responses-compatible model provider endpoint
  • Long-running automated policy-review workload

Sanitized Timeline

  • T+2h30m00s: MCP rule approved; gateway emits policy version 43 as approved.
  • T+2h30m02s: sandbox reports policy version 43 load status failed.
  • T+2h30m05s: the next model request disconnects.
  • Following minutes: policy versions 44, 45, and 46 also report load status failed.
  • T+2h45m: execution relay closes and the sandbox stops.

No credentials, private endpoint names, model identifiers, or private repository details are included in this report.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the ApproveDraftChunk approval path and trace how gateway policy status, sandbox load status, and load_error are recorded. Reproduce approval followed by a sandbox load failure, then verify that status, rollback, subsequent composition, and retained error reporting match the acceptance criteria; the payload names no specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.