[All Platforms][Onboard] Portable profile's hardcoded registry IP has no recovery when Docker IPAM state gets stuck
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
`nemoclaw onboard --experimental-profile portable` always requests the exact same hardcoded Docker network name, subnet, and container IP for its "managed portable registry" container (`PORTABLE_DOCKER_NETWORK_NAME` = the default docker-driver gateway network name; `PORTABLE_DOCKER_NETWORK_SUBNET` = "10.87.0.0/24"; `PORTABLE_REGISTRY_IP` = "10.87.0.3" -- `src/lib/onboard/experimental/portable-profile.ts`). If Docker's daemon-level IPAM ever ends up with a stale/orphaned reservation for that exact IP (a well-known general Docker failure mode after an abruptly-killed or force-removed container -- e.g. an interrupted onboard attempt, a host reboot mid-run, or any ungraceful teardown), onboarding fails hard and permanently with a raw Docker error, and there is no NemoClaw-level detection, remediation guidance, or automatic retry/cleanup path. The stale reservation can reference a container ID that no longer exists in `docker ps -a` at all, and survives a full `systemctl restart docker` (confirmed live -- the daemon restart produced a fresh network ID but the exact same IP+container-ID error recurred identically on the next onboard attempt).
This is related-but-distinct from the already-closed #9707 (netavark rejecting the OLD hardcoded subnet 169.254.1.0/24 -- fixed by migrating to the current 10.87.0.0/24). The subnet migration fixed netavark's validation rejection, but the underlying design pattern -- a single fixed IP/subnet reused across all onboard attempts, with no fallback when that fixed resource is unavailable for any reason -- remains, and reproduces under a different trigger (Docker IPAM staleness rather than netavark policy).
Platform scope: Reproduced on DGX Spark (aarch64) only; other platforms not tested this pass. Root cause is Docker/NemoClaw's own IPAM interaction, not hardware-specific -- expected to reproduce on any host running the portable profile after any ungraceful container teardown.
Regression: Unknown -- earlier versions not tested this pass. This is an experimental profile with active recent changes (see #9707).
OpenShell issue: No -- root cause is in NemoClaw's own onboard code (hardcoded IP/network constants with no stuck-state recovery), not the OpenShell CLI.
## Environment
```text
Device: DGX Spark (GB10)
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Docker: 29.2.1
Podman: 5.7.0 (rootless, exact version required by the portable profile's own preflight)
NemoClaw: v0.0.121
```
## Steps to Reproduce
1. On a host where `nemoclaw onboard --experimental-profile portable` has been attempted before and interrupted/force-killed at least once (a completely normal occurrence -- Ctrl+C, session timeout, retry after a preflight failure, etc.), leaving Docker's daemon-level IPAM state with a stale reservation for 10.87.0.3 tied to a now-nonexistent container ID.
2. Run `nemoclaw onboard --experimental-profile portable --fresh --agent hermes --name --yes` (or `--resume`).
3. Observe the failure. Confirm via `docker ps -a` (empty / no matching container) and `docker network ls` (no network currently holding that subnet) that the referenced container genuinely does not exist.
4. Run `sudo systemctl restart docker` and retry step 2 -- the identical error (same IP, same stale container ID) recurs immediately.
## Expected Result
Onboarding either succeeds by detecting and cleaning up the stale IPAM reservation itself, or fails with actionable NemoClaw-level guidance identifying the stuck resource and a concrete recovery step -- not a raw, unexplained Docker daemon error that requires deep Docker-internals knowledge to diagnose.
## Actual Result
```text
Error: Starting the managed portable registry failed: docker: Error response from daemon: IPAM error: requested ip address 10.87.0.3 is already allocated to container ID 49f6af196bc7cf81b292b3275984727db24c35c2ed53adf35851f680d19c18d8
```
`docker inspect 49f6af196bc7` returns "no such object". `docker ps -a` and `docker network ls` show no trace of any container or network holding that address. The error is identical across a `--fresh` attempt, a `--resume` attempt, and a fresh attempt after a full `systemctl restart docker` -- proving it is not resolvable through any NemoClaw-level or normal Docker-level action available to the user.
## Logs
Not captured beyond the exact CLI error output shown above (reproduced 3 times with identical text across --fresh, --resume, and post-daemon-restart attempts).
Contributor guide
Research direction
Start in src/lib/onboard/experimental/portable-profile.ts and trace the portable-profile onboarding path that creates and starts the managed registry. Run the listed nemoclaw onboard reproduction, then inspect how Docker network and fixed 10.87.0.3 allocation failures are surfaced. Done means the stale allocation is handled or the CLI reports a concrete NemoClaw recovery path instead of the raw daemon error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, typescript
- Domain
- cli, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100