NVIDIA / NVIDIA/NemoClaw

[Linux][Sandbox] Podman-driven sandbox container start fails with "timeout after 30s" even when the image is already cached

Open
#10,958 0 comments 0 reactions 0 assignees View on GitHub
area: onboarding area: sandbox NV QA platform: container platform: linux
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

When onboarding uses the native Podman managed-image path (`NEMOCLAW_GATEWAY_RUNTIME=podman`), sandbox container creation frequently fails with `timeout after 30s` during the container-start step, even when the sandbox image is already present locally (no pull needed). Root cause: the Podman API client's `start_container()` call uses a hardcoded 30-second HTTP timeout with no override path.

Platform scope: Reproduced on Linux only; other platforms not tested.
Regression: No — new functionality in this release (native Podman managed-image onboarding path).
OpenShell issue: No — the timeout constant lives in the OpenShell Podman driver crate (`crates/openshell-driver-podman/src/client.rs`), which is consumed by NemoClaw's onboarding flow.

## Environment

```text
Device: Linux server (rootless Podman host)
OS: Linux
Architecture: x86_64 (linux/amd64)
Node.js: v22.23.2
npm: Not captured
Docker: Not captured (installed and running; used for the gateway launcher, not the sandbox container runtime in this path)
OpenShell CLI: 0.0.106
NemoClaw: v0.0.119
OpenClaw: 2026.7.1
```

## Steps to Reproduce

1. On a Linux host with a healthy rootless Podman service (verified: `podman info` succeeds, cgroups v2, bridge networking and DNS working) and valid provider credentials configured, run:
```bash
export NEMOCLAW_GATEWAY_RUNTIME=podman
nemoclaw onboard --fresh --name {sandbox} --no-gpu --yes-i-accept-third-party-software
```
2. Ensure the managed sandbox image (`ghcr.io/nvidia/nemoclaw/openclaw-sandbox`) is already present locally from a prior run, so this attempt does not need to pull it.
3. Observe the "[6/8] Creating sandbox" step through to completion or failure.

## Expected Result

The sandbox container starts successfully and onboarding proceeds past "[6/8] Creating sandbox" to completion, matching the documented native Podman managed-image onboarding path.

## Actual Result

Sandbox creation failed (exit 1):

```text
Error: x code: 'Internal error', message: "create sandbox failed: timeout after 30s"
```

Reproduced 3 times in a row on one host (image already cached each time — no pull phase involved) with all three attempts failing at almost exactly 30 seconds. On a second host, the same flow succeeded once, but only because most of the elapsed time (~51s total) was spent on a separate first-time image pull phase that precedes the container-start call and is not bound by the same 30-second limit — the actual `start_container()` call itself was not stress-tested there the same way.

## Logs

```text
INFO request{method=POST path="/openshell.v1.OpenShell/CreateSandbox" ...}:driver{... driver.name=podman ...}: openshell_driver_podman::driver: Creating sandbox container sandbox_id={id} sandbox_name={sandbox}
WARN request{method=POST path="/openshell.v1.OpenShell/CreateSandbox" ...}:driver{... driver.name=podman ...}: openshell_driver_podman::driver: Failed to start container; cleaning up sandbox_name={sandbox} error=timeout after 30s
```

## Root Cause (from public NVIDIA/OpenShell source)

File: `crates/openshell-driver-podman/src/client.rs`

```rust
const API_TIMEOUT: Duration = Duration::from_secs(30);

pub async fn start_container(&self, name: &str) -> Result<(), PodmanApiError> {
validate_name(name)?;
self.request_ok(hyper::Method::POST, &format!("/libpod/containers/{name}/start"), None).await
}

async fn request_ok(&self, method, path, body) -> Result<(), PodmanApiError> {
let (status, bytes) = self.request(method, path, body, API_TIMEOUT).await?;
...
}
```

`start_container()` routes through `request_ok()`, which always passes the hardcoded `API_TIMEOUT` constant with no parameter to override it. By contrast, `stop_container()` in the same file computes a dynamic timeout (`timeout_secs + 5`) from a caller-supplied grace period. `start_container()` has no equivalent mechanism, so any container whose actual startup exceeds 30 seconds fails outright regardless of host load or image size, instead of waiting longer or retrying.

## Proposed Fix

Give `start_container()` a caller-configurable timeout (or a larger default), similar to how `stop_container()` already computes its timeout dynamically, so container-start latency variance across hosts does not cause spurious onboarding failures.

Contributor guide

Open the contributing guide

Research direction

Start in crates/openshell-driver-podman/src/client.rs by tracing start_container() through request_ok() and comparing its fixed API_TIMEOUT with stop_container()'s caller-supplied timeout. Then locate the start_container() callers and relevant tests, if present. Done means the native Podman onboarding path no longer fails solely because container startup exceeds the current hardcoded limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.