NVIDIA / NVIDIA/OpenShell

VM sandbox SSH session disconnects with broken pipe

Open
#2,587 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

state:triage-needed
Dominant language
Rust
Stars
8.7k
Forks
1.3k
Avg merge
2d 11h
Merged PRs (30d)
253

Description

Agent Diagnostic
  • Skills loaded: create-github-issue (repo skill), direct code investigation of crates/openshell-supervisor-process,
    openshell-supervisor-network, openshell-driver-vm, openshell-sandbox, openshell-server
  • Latest release checked: OpenShell: 0.0.96 on 6.18.33.2-microsoft-standard-WSL2
  • Findings: SSH session to a VM-driver sandbox completes a full handshake and shell setup (pty_request/env_request/shell_request all
    succeed), then dies ~14.5s later with write to target failed: Broken pipe (os error 32) on the relay bridge to
    unix:/run/openshell/ssh.sock. No timeout/keepalive/rekey constant in supervisor_session.rs, the gateway's relay-pending timeouts
    (grpc/sandbox.rs), or the embedded SSH server config (ssh.rs) matches this window — the only 10s/15s timeouts found gate
    pre-establishment (waiting for the relay to be claimed), not an already-bridging channel. The sandbox supervisor process itself is
    confirmed still alive well past the failure (background Fetching inference route bundle polling continues uninterrupted every ~5s for
    the following minute), which rules out a whole-process/VM-guest death and scopes the bug to a single relay bridge/channel.
  • Remaining reason for filing: an interactive SSH session to a keep-alive entrypoint (tail -f /dev/null) should stay open until the
    user disconnects; a ~15s spontaneous teardown with no matching timeout in the code and no client-side activity in that window indicates
    a real defect in per-channel session/lifecycle handling under the VM driver, not expected behavior.
Description

Actual behavior: After openshell sandbox create (VM compute driver), the auto-connected SSH session completes handshake and shell
setup successfully, then disconnects ~14-15 seconds later with:
Warning: Permanently added 'sandbox' (ED25519) to the list of known hosts.
Connection to sandbox closed by remote host.
Connection to sandbox closed.
client_loop: send disconnect: Broken pipe
The gateway logs ForwardTcp: inbound stream ended and the sandbox logs a relay bridge failure (write to target failed: Broken pipe (os error 32)) at the same moment, while the sandbox process itself keeps running normally afterward.

Expected behavior: The SSH session should remain open indefinitely (entrypoint is tail -f /dev/null, a keep-alive command) until
the user disconnects.

Reproduction Steps
  1. Use a gateway.toml with the VM compute driver: 
    [openshell.gateway]
compute_drivers = ["vm"]
log_level = "debug"

[openshell.drivers.vm]
mem_mib = 4096
vcpus = 2
default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:fffb6b2"
bootstrap_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:fffb6b2"
  2. Run openshell -v sandbox create.
  3. Observe the SSH session connect, then disconnect ~14-15 seconds later.
Environment

Environment

  • OS: 6.18.33.2-microsoft-standard-WSL2
  • OpenShell: 0.0.96
Logs

cli

openshell -v sandbox create

Created sandbox: dreamy-halibut

✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
✓ Sandbox allocated (0s)
Warning: Permanently added 'sandbox' (ED25519) to the list of known hosts.
Connection to sandbox closed by remote host.
Connection to sandbox closed.
client_loop: send disconnect: Broken pipe

gateway/sandbox

12:35:58 gateway INFO  minted sandbox JWT
12:35:58 gateway INFO  CreateSandbox request completed successfully
12:35:59 sandbox INFO  Starting sandbox command=["tail", "-f", "/dev/null"]
12:35:59 sandbox INFO  Fetching sandbox policy via gRPC endpoint=https://host.openshell.internal:17670/ sandbox_id=cbea7e3a-fda6-4e20-90e4-6816a3dc3984
12:35:59 gateway DEBUG GetSandboxConfig: no policy configured, returning empty response
12:35:59 gateway INFO  UpdateConfig: backfilled spec.policy from sandbox-discovered policy
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway INFO  GetSandboxProviderEnvironment request completed successfully
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway INFO  ReportPolicyStatus: sandbox reported policy load result
12:35:59 gateway INFO  supervisor session: accepted
12:36:00 gateway INFO  supervisor session: relay opened successfully
12:36:00 gateway INFO  relay stream: claimed pending relay, bridging
12:35:59 sandbox OCSF  CONFIG:DISCOVERY [INFO] Server returned no policy; attempting local discovery
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Loaded sandbox policy from container disk [path:/etc/openshell/policy.yaml]
12:35:59 sandbox INFO  Creating OPA engine from proto policy data
12:35:59 sandbox INFO  Configured OPA runtime binary identity mode require_binary_identity=true source=proto
12:35:59 sandbox OCSF  CONFIG:CONFIGURED [INFO] OPA runtime binary identity mode configured [source:proto require_binary_identity:true] [source:proto]
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: claude_code.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: github_rest_api.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: github_ssh_over_https.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Fetched provider environment [env_count:0]
12:35:59 sandbox OCSF  CONFIG:CREATING [INFO] Creating network namespace [ns:sandbox-61660b7c host_veth:veth-h-61660b7c sandbox_veth:veth-s-61660b7c]
12:35:59 sandbox OCSF  CONFIG:CREATED [INFO] Network namespace created [ns:sandbox-61660b7c host_ip:10.200.0.1 sandbox_ip:10.200.0.2]
12:35:59 sandbox WARN  non-required nft command failed in namespace (continuing) command=add rule inet openshell_bypass output tcp flags syn limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid error=Error: syntax error, unexpected colon, expecting end of file or newline or semicolon
add rule inet openshell_bypass output tcp flags syn limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid
                                                                                                             ^ netns=sandbox-61660b7c
12:35:59 sandbox WARN  non-required nft command failed in namespace (continuing) command=add rule inet openshell_bypass output meta l4proto udp limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid error=Error: syntax error, unexpected colon, expecting end of file or newline or semicolon
add rule inet openshell_bypass output meta l4proto udp limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid
                                                                                                                ^ netns=sandbox-61660b7c
12:35:59 sandbox OCSF  CONFIG:INSTALLED [INFO] Bypass detection rules installed [ns:sandbox-61660b7c]
12:35:59 sandbox OCSF  CONFIG:ENABLED [INFO] TLS termination enabled: ephemeral CA generated
12:35:59 sandbox INFO  Fetching inference route bundle from gateway endpoint=https://host.openshell.internal:17670/
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Loaded inference route bundle [route_count:0 revision:d1fba762150c532c]
12:35:59 sandbox OCSF  CONFIG:WAITING [INFO] Inference route bundle is empty; keeping routing enabled and waiting for refresh
12:35:59 sandbox OCSF  CONFIG:ENABLED [INFO] Inference routing enabled with local execution [route_count:0]
12:35:59 sandbox OCSF  NET:LISTEN [INFO] 10.200.0.1:3128
12:35:59 sandbox WARN  host.openshell.internal maps to a non-link-local IP; trusted-gateway SSRF exemption disabled ip=192.168.127.254
12:35:59 sandbox OCSF  CONFIG:VALIDATED [INFO] Validated 'sandbox' user exists in image
12:35:59 sandbox OCSF  CONFIG:VALIDATED [INFO] Validated 'sandbox' group exists in image
12:35:59 sandbox WARN  runtime cgroup pids.max is unavailable (No such file or directory (os error 2)); configure the compute driver or container runtime to enforce a PID limit
12:35:59 sandbox OCSF  SSH:LISTEN [INFO]
12:35:59 sandbox OCSF  LIFECYCLE:INSTALL [INFO] OpenShell Sandbox Supervisor success
12:35:59 sandbox INFO  supervisor session task spawned
12:35:59 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:35:59 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Acknowledged initial policy revision as loaded [version:1] [hash:27380a8cbc68351c4aa9a60cdfebe905ec647600bb2e5d527e13f1a298998792]
12:35:59 sandbox OCSF  NET:OPEN [INFO] host.openshell.internal:17670
12:35:59 sandbox OCSF  NET:OPEN [INFO] [msg:ssh relay open (channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c, target=unix:/run/openshell/ssh.sock)]
12:35:59 sandbox OCSF  SSH:OPEN [INFO] ALLOWED
12:36:00 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:36:04 sandbox WARN  Entrypoint PID never published; binary symlink resolution skipped. Policy binary paths will be matched literally.
12:36:09 gateway DEBUG GetSandboxConfig served from policy history
12:36:14 gateway DEBUG ForwardTcp: inbound stream ended
12:36:14 sandbox OCSF  NET:FAIL [LOW] [msg:ssh relay bridge failed (channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c, target=unix:/run/openshell/ssh.sock): write to...]
12:36:14 sandbox WARN  supervisor session: relay bridge failed channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c error=write to target failed: Broken pipe (os error 32) sandbox_id=cbea7e3a-fda6-4e20-90e4-6816a3dc3984
12:36:19 gateway DEBUG GetSandboxConfig served from policy history
12:36:26 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:26 gateway DEBUG GetSandboxConfig served from policy history
12:35:58 gateway INFO  minted sandbox JWT
12:35:58 gateway INFO  CreateSandbox request completed successfully
12:35:59 sandbox INFO  Starting sandbox command=["tail", "-f", "/dev/null"]
12:35:59 sandbox INFO  Fetching sandbox policy via gRPC endpoint=https://host.openshell.internal:17670/ sandbox_id=cbea7e3a-fda6-4e20-90e4-6816a3dc3984
12:35:59 gateway DEBUG GetSandboxConfig: no policy configured, returning empty response
12:35:59 gateway INFO  UpdateConfig: backfilled spec.policy from sandbox-discovered policy
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway INFO  GetSandboxProviderEnvironment request completed successfully
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway DEBUG GetSandboxConfig served from policy history
12:35:59 gateway INFO  ReportPolicyStatus: sandbox reported policy load result
12:35:59 gateway INFO  supervisor session: accepted
12:36:00 gateway INFO  supervisor session: relay opened successfully
12:36:00 gateway INFO  relay stream: claimed pending relay, bridging
12:35:59 sandbox OCSF  CONFIG:DISCOVERY [INFO] Server returned no policy; attempting local discovery
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Loaded sandbox policy from container disk [path:/etc/openshell/policy.yaml]
12:35:59 sandbox INFO  Creating OPA engine from proto policy data
12:35:59 sandbox INFO  Configured OPA runtime binary identity mode require_binary_identity=true source=proto
12:35:59 sandbox OCSF  CONFIG:CONFIGURED [INFO] OPA runtime binary identity mode configured [source:proto require_binary_identity:true] [source:proto]
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: claude_code.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: github_rest_api.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:VALIDATED [MED] L7 policy validation warning: github_ssh_over_https.endpoints[0]: 'tls: terminate' is deprecated; TLS termination is now automatic. Use 'tls: skip' to disable.
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Fetched provider environment [env_count:0]
12:35:59 sandbox OCSF  CONFIG:CREATING [INFO] Creating network namespace [ns:sandbox-61660b7c host_veth:veth-h-61660b7c sandbox_veth:veth-s-61660b7c]
12:35:59 sandbox OCSF  CONFIG:CREATED [INFO] Network namespace created [ns:sandbox-61660b7c host_ip:10.200.0.1 sandbox_ip:10.200.0.2]
12:35:59 sandbox WARN  non-required nft command failed in namespace (continuing) command=add rule inet openshell_bypass output tcp flags syn limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid error=Error: syntax error, unexpected colon, expecting end of file or newline or semicolon
add rule inet openshell_bypass output tcp flags syn limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid
                                                                                                             ^ netns=sandbox-61660b7c
12:35:59 sandbox WARN  non-required nft command failed in namespace (continuing) command=add rule inet openshell_bypass output meta l4proto udp limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid error=Error: syntax error, unexpected colon, expecting end of file or newline or semicolon
add rule inet openshell_bypass output meta l4proto udp limit rate 5/second burst 10 packets log prefix openshell:bypass:sandbox-61660b7c: flags skuid
                                                                                                                ^ netns=sandbox-61660b7c
12:35:59 sandbox OCSF  CONFIG:INSTALLED [INFO] Bypass detection rules installed [ns:sandbox-61660b7c]
12:35:59 sandbox OCSF  CONFIG:ENABLED [INFO] TLS termination enabled: ephemeral CA generated
12:35:59 sandbox INFO  Fetching inference route bundle from gateway endpoint=https://host.openshell.internal:17670/
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Loaded inference route bundle [route_count:0 revision:d1fba762150c532c]
12:35:59 sandbox OCSF  CONFIG:WAITING [INFO] Inference route bundle is empty; keeping routing enabled and waiting for refresh
12:35:59 sandbox OCSF  CONFIG:ENABLED [INFO] Inference routing enabled with local execution [route_count:0]
12:35:59 sandbox OCSF  NET:LISTEN [INFO] 10.200.0.1:3128
12:35:59 sandbox WARN  host.openshell.internal maps to a non-link-local IP; trusted-gateway SSRF exemption disabled ip=192.168.127.254
12:35:59 sandbox OCSF  CONFIG:VALIDATED [INFO] Validated 'sandbox' user exists in image
12:35:59 sandbox OCSF  CONFIG:VALIDATED [INFO] Validated 'sandbox' group exists in image
12:35:59 sandbox WARN  runtime cgroup pids.max is unavailable (No such file or directory (os error 2)); configure the compute driver or container runtime to enforce a PID limit
12:35:59 sandbox OCSF  SSH:LISTEN [INFO]
12:35:59 sandbox OCSF  LIFECYCLE:INSTALL [INFO] OpenShell Sandbox Supervisor success
12:35:59 sandbox INFO  supervisor session task spawned
12:35:59 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:35:59 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:35:59 sandbox OCSF  CONFIG:LOADED [INFO] Acknowledged initial policy revision as loaded [version:1] [hash:27380a8cbc68351c4aa9a60cdfebe905ec647600bb2e5d527e13f1a298998792]
12:35:59 sandbox OCSF  NET:OPEN [INFO] host.openshell.internal:17670
12:35:59 sandbox OCSF  NET:OPEN [INFO] [msg:ssh relay open (channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c, target=unix:/run/openshell/ssh.sock)]
12:35:59 sandbox OCSF  SSH:OPEN [INFO] ALLOWED
12:36:00 sandbox OCSF  FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" [type:landlock-unavailable confidence:high]
12:36:04 sandbox WARN  Entrypoint PID never published; binary symlink resolution skipped. Policy binary paths will be matched literally.
12:36:09 gateway DEBUG GetSandboxConfig served from policy history
12:36:14 gateway DEBUG ForwardTcp: inbound stream ended
12:36:14 sandbox OCSF  NET:FAIL [LOW] [msg:ssh relay bridge failed (channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c, target=unix:/run/openshell/ssh.sock): write to...]
12:36:14 sandbox WARN  supervisor session: relay bridge failed channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c error=write to target failed: Broken pipe (os error 32) sandbox_id=cbea7e3a-fda6-4e20-90e4-6816a3dc3984
12:36:19 gateway DEBUG GetSandboxConfig served from policy history
12:36:26 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:26 gateway DEBUG GetSandboxConfig served from policy history
12:36:28 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:28 gateway DEBUG GetSandboxConfig served from policy history
12:36:28 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:30 gateway DEBUG GetSandboxConfig served from policy history
12:36:30 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:30 gateway DEBUG GetSandboxConfig served from policy history
12:36:30 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:32 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:32 gateway DEBUG GetSandboxConfig served from policy history
12:36:32 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:34 gateway DEBUG GetDraftPolicy: served draft chunks
12:36:34 gateway DEBUG GetSandboxConfig served from policy history

rootfs-console.log

  2026-08-02T12:36:00.013Z DEBUG log: handler.shell_request ChannelId(2)
  2026-08-02T12:36:00.016Z DEBUG log: channel_success ... (shell established)
  2026-08-02T12:36:04.702Z WARN openshell_supervisor_network::run: Entrypoint PID never published; binary symlink resolution skipped.
  2026-08-02T12:36:14.514Z OCSF NET:FAIL [msg:ssh relay bridge failed (channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c,
  target=unix:/run/openshell/ssh.sock): write to...]
  2026-08-02T12:36:14.516Z WARN openshell_supervisor_process::supervisor_session: supervisor session: relay bridge failed
  channel_id=22fb2142-c378-4efb-bab8-4efd718e8e7c error=write to target failed: Broken pipe (os error 32)
  2026-08-02T12:36:19.725Z DEBUG openshell_core::grpc_client: Fetching inference route bundle   <- background task confirmed still alive
  after the failure, repeating every ~5s through 12:37:10+

  Full raw debug logs (gateway + rootfs-console, including h2/TLS trace-level noise) available on request.
Agent-First Checklist
  • I pointed my agent at the repo and had it investigate this issue
  • I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
  • I checked the latest OpenShell release and either reproduced the issue there or explained why I cannot upgrade/test it
  • I searched existing issues for possible duplicates or explained why I could not
  • My agent could not resolve this — the diagnostic above explains why

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/openshell-supervisor-process, especially supervisor_session.rs, and trace the SSH relay bridge to unix:/run/openshell/ssh.sock. Compare that lifecycle with grpc/sandbox.rs and the embedded SSH configuration in ssh.rs, then reproduce with the VM driver and tail -f /dev/null. Done means the established session remains open until the client disconnects without a relay bridge failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.