bug(podman): sandbox remains Ready after relay or service forward becomes unavailable

Summary

This issue reports a possible OpenShell relay and forwarding lifecycle problem observed with the Podman compute driver. It is not specific to AI Guardian. AI Guardian exposed the problem because it uses openshell forward service to reach a daemon running inside the sandbox.

The failure is currently time-dependent and has not been reduced to a deterministic trigger.

Environment

• OpenShell gateway: 0.0.116
• Embedded sandbox supervisor in both affected containers: openshell-sandbox 0.0.116
• Compute driver: rootless Podman
• Host: Fedora Linux with SELinux enforcing, x86_64
• Two long-lived sandboxes

No usernames, hostnames, home directories, repository paths, container IDs, sandbox UUIDs, credentials, or private URLs are included here.

Observed behavior

After several hours or overnight:

• openshell sandbox list continued to report both sandboxes as Ready.
• podman ps showed both corresponding containers as Up (healthy).
• openshell sandbox exec and interactive connection attempts failed.
• Host-side openshell forward service processes were no longer running.
• openshell forward list showed stale or dead forward records.
• No local listening socket remained for the recorded forward ports.
• The gateway remained healthy, but the sandbox control and service-forward paths were unusable.

No OOM kill or unhealthy container state was observed.

Minimal OpenShell-only observation harness

The following example does not use AI Guardian. It creates a long-lived sandbox containing a simple loopback HTTP service, then forwards that service through OpenShell. Use a disposable sandbox and replace the local port if 43152 is already in use.

Terminal A:

openshell sandbox create \
  --name openshell-forward-repro \
  --from base \
  --no-auto-providers \
  -- \
  sh -lc 'exec python3 -m http.server 63152 --bind 127.0.0.1'

Terminal B, after the sandbox is ready:

openshell sandbox exec --name openshell-forward-repro --no-tty -- \
  sh -lc 'curl --fail --silent --show-error http://127.0.0.1:63152/'

Terminal C, start the host-side service forward and leave it running:

openshell forward service openshell-forward-repro \
  --target-host 127.0.0.1 \
  --target-port 63152 \
  --local 127.0.0.1:43152

In another shell, periodically test the forward and the sandbox control path:

curl --fail --silent --show-error http://127.0.0.1:43152/
openshell sandbox exec --name openshell-forward-repro --no-tty -- true
openshell sandbox list
openshell forward list

For a longer observation, leave the sandbox and forward running for several hours or overnight, then repeat the checks. The failure is considered reproduced if the container remains healthy and the sandbox remains Ready, but the forward process exits or the local port stops listening and subsequent exec or connection requests fail.

Sanitized state evidence

Gateway: openshell
Status: healthy
Version: 0.0.116
Compute drivers: podman (0.0.116)

NAME       PHASE
sandbox-a  Ready
sandbox-b  Ready

Forward records for the affected sandboxes: dead
Podman containers for the affected sandboxes: Up (healthy)

Relevant sanitized logs

The affected run included messages similar to:

[gateway] supervisor session: relay opened successfully
[gateway] relay stream: claimed pending relay, bridging
[sandbox] supervisor session: relay bridge failed
[sandbox] write to target failed: Broken pipe (os error 32)
[gateway] ForwardTcp: inbound stream ended
[sandbox] supervisor session: stream closed by supervisor
[sandbox] supervisor session: ended

During one manual stop/start transition, the sandbox log also contained:

WARN openshell_supervisor_process::netns: Failed to delete network namespace

This line may be secondary to the restart and is not asserted as the root cause.

Relation to existing reports

• #2372 describes the SSH accept loop exiting after transient errors while the sandbox remains Ready.
• #2337 describes a similar proxy accept-loop failure under file-descriptor exhaustion.
• #2587 describes broken-pipe relay failures, although that report is VM-specific.

The 0.0.116 supervisor source contains retry/backoff handling for transient SSH accept errors and monitoring for an unexpected SSH accept-loop exit:

• SSH supervisor source at v0.0.116
• Sandbox lifecycle source at v0.0.116

Therefore, this may be a different failure in the host-side openshell forward service or in the relay lifecycle. The captured logs do not contain EMFILE, so file-descriptor exhaustion is only a hypothesis and is not confirmed.

Expected behavior

• A sandbox reported as Ready should have working exec, interactive connection, and required service-forwarding paths.
• If a required relay or forward dies, OpenShell should either recover it or mark the sandbox degraded/error instead of leaving it Ready.
• A detached or long-lived forward should have an actionable failure status and a supported recovery or reconciliation mechanism.
• Restarting a sandbox should restore the complete control and forwarding path, or identify exactly which component could not be restored.

Questions

1. Can the gateway or Podman supervisor detect and recover a lost relay or service-forward path?
2. Is there a health signal that distinguishes a healthy Podman container from a sandbox whose SSH or forwarding relay is unusable?
3. Should openshell forward service be supervised or automatically reconciled when the underlying relay closes?
4. Are there known OpenShell 0.0.116 issues involving long-lived Podman sandboxes, repeated ForwardTcp traffic, or forward processes exiting without changing the sandbox phase?

Suggested diagnostics

If this is reproduced, please collect only sanitized output from:

openshell version
openshell gateway info
openshell sandbox list
openshell forward list
openshell logs openshell-forward-repro --since 30m
podman ps -a --filter label=openshell.managed=true

Please redact local usernames, hostnames, filesystem paths, container IDs, sandbox UUIDs, credentials, and private endpoints before attaching logs.

Start with the Podman reproduction harness and the referenced crates/openshell-supervisor-process/src/ssh.rs and crates/openshell-sandbox/src/lib.rs at v0.0.116, then collect the listed sanitized gateway, sandbox, forward, and Podman state. Compare relay, exec, and forward lifecycles across a failure and a restart. Done means the failure is deterministic or narrowed to a component, with recovery or degraded-state behavior defined and covered by evidence or tests.