NVIDIA / NVIDIA/NemoClaw

[Ubuntu 25.04][Onboard] native Podman onboarding fails at the bootstrap original-container handoff and its rollback also fails, leaving no sandbox

Open
#11,967 0 comments 0 reactions 0 assignees View on GitHub
needs: triage NV QA UAT
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

Onboarding a sandbox through the native Podman runtime (`NEMOCLAW_GATEWAY_RUNTIME=podman`) never completes. The sandbox image builds, sandbox creation starts, and the run then aborts inside NemoClaw's own Podman bootstrap:

```text
Podman bootstrap original-container handoff removal failed with status 2.
Managed bootstrap rollback requires attention: Podman bootstrap original-container
rollback start failed with status 125.
```

Both halves fail: the handoff step that should remove the bootstrap's original container fails, and the rollback that exists to clean up after that failure fails too. The message says rollback "requires attention" but gives the operator no command, no container identity, and no state to inspect.

No sandbox is produced. `status` reports nothing for the requested name, and the only residue is a container left in Exited state. The whole native-Podman managed-image workflow is therefore unavailable on this build.

This is not leftover-state contamination. The failure was reproduced twice: once on a host that had prior Podman activity, and again after removing every container, pruning the Podman system to zero containers, and using a previously unused gateway port and a previously unused sandbox name. Both runs failed identically at the same step with the same two status codes.

- **Platform scope:** Reproduced on Ubuntu 25.04 x86_64 with rootless Podman 5.7.0; other platforms not tested this pass.
- **Regression:** Unknown — earlier builds not tested this pass.
- **OpenShell issue:** No — the failure is raised by NemoClaw's managed Podman bootstrap during sandbox creation, before any sandbox is registered.

## Environment

```text
Device: x86_64 server with NVIDIA A100-SXM4-40GB
OS: Ubuntu 25.04
Architecture: x86_64
Podman: 5.7.0 (client and server), rootless, user socket active
Docker: installed but not used for this path
OpenShell CLI: openshell 0.0.116
NemoClaw: nemoclaw v0.0.126
Agent: OpenClaw
Provider: NVIDIA cloud inference
```

## Steps to Reproduce

1. On a Linux host with rootless Podman active and cgroups v2, confirm the Podman user socket is running and that `DOCKER_HOST`, `DOCKER_CONTEXT` and `CONTAINER_HOST` are unset.
2. Remove all Podman containers and prune, so the run starts from zero containers.
3. Choose a gateway port that has never been used on this host.
4. Run, with a previously unused sandbox name:

```bash
NEMOCLAW_GATEWAY_RUNTIME=podman nemoclaw onboard --no-sandbox-gpu --name {sandbox} \
--fresh --non-interactive --yes --yes-i-accept-third-party-software
```

5. Read the tail of the output and the exit code.
6. Run `nemoclaw {sandbox} status` and `podman ps -a`.

## Expected Result

Onboarding accepts the native Podman selection without requiring a Docker daemon and completes, leaving the sandbox Ready.

If the bootstrap handoff genuinely cannot complete, the rollback should succeed and leave the host clean, and the error should identify the container it could not hand off or remove and what the operator should do.

## Actual Result

The image builds and sandbox creation begins:

```text
Creating sandbox '{sandbox}' (this takes a few minutes on first run)...
Updated provider nvidia-prod
Building sandbox image...
Still building sandbox image... (45s elapsed)
Still working on Sandbox creation... (76s elapsed)
```

Then the run aborts:

```text
Error: Podman bootstrap original-container handoff removal failed with status 2.
Managed bootstrap rollback requires attention: Podman bootstrap original-container
rollback start failed with status 125.

Onboarding did not finish. Resume from the step that failed with:
nemoclaw onboard --resume

exit 1
```

Afterwards:

```text
nemoclaw {sandbox} status no phase reported
podman ps -a one container in Exited (137) state
agent request exit 1, no response
```

Two runs, same result:

| run | starting state | result |
|---|---|---|
| 1 | host with prior Podman activity, previously used port | handoff status 2 / rollback status 125 |
| 2 | 0 containers after prune, unused port, unused name | handoff status 2 / rollback status 125 |

## Logs

```text
onboard exit code 1
"original-container handoff removal failed" present in both runs
"rollback start failed" present in both runs
podman version client / server 5.7.0 / 5.7.0
podman user socket active
containers before run 2 0
free disk before run 2 38G
```

## Related issues / not duplicate of

No existing report covers the bootstrap original-container handoff or its rollback failing.

Nearby Podman reports, none of them this failure:

- **#11716** — Podman onboarding rejected Podman's own default socket folder permissions. That is a preflight permission check and is already fixed; here preflight passes and the failure is during sandbox creation.
- **#11721** — the installer requires Docker to be running. That is install-time; this is onboarding on a host where installation already succeeded.
- **#10266** — portable-profile image build failed on Podman because of `ADD --checksum`. That is an image-build syntax failure; here the image builds successfully and the failure is in the bootstrap handoff afterwards.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the native Podman path with the provided nemoclaw onboard command, then inspect the bootstrap handoff and rollback entry points that emit the reported errors. Compare the failed container state with podman ps -a and nemoclaw {sandbox} status; done means onboarding completes with a Ready sandbox, or rollback leaves the host clean and reports actionable container details.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.