NVIDIA / NVIDIA/NemoClaw

[DGX Spark][Onboard] onboarding cannot allocate a dashboard port and aborts with "could not prove OpenShell forward ownership" while most of the documented port range is free

Open
#11,979 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

On DGX Spark, onboarding aborts before creating the sandbox:

```text
Error: Cannot allocate dashboard port: NemoClaw could not prove OpenShell forward ownership.
```

Nothing about the host explains the refusal. The documented dashboard range has nine of its eleven ports free, the gateway port is one that has never been used on the host and has no state directory, and the sandbox name is new. The message names no port, no competing owner, and no action the operator can take.

The failure is not specific to a profile or an agent: it reproduces with a plain `onboard` using the default agent and cloud inference, with no managed local-inference profile involved.

Scope observed so far: both DGX Spark hosts available for this pass reproduce it. The same build on three non-Spark Linux hosts onboarded successfully many times during the same session, so the failure appears specific to these Spark hosts rather than to the build in general. What distinguishes them is that they carry other users' sandboxes that already hold some dashboard ports, but free ports remain in the range.

- **Platform scope:** Reproduced on two DGX Spark (GB10) hosts running Ubuntu 24.04 aarch64; non-Spark Linux hosts on the same build did not reproduce it this pass.
- **Regression:** Unknown — earlier builds not tested for this path this pass.
- **OpenShell issue:** Unclear from the outside. The message attributes the failure to an OpenShell forward-ownership proof, but it is NemoClaw's onboarding that refuses; the OpenShell gateway itself is healthy and `openshell forward list` reports no active forwards.

## Environment

```text
Device: NVIDIA DGX Spark (GB10), two separate hosts
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Docker: Docker version 29.2.1
OpenShell CLI: openshell 0.0.116
NemoClaw: nemoclaw v0.0.126
Agent: OpenClaw
Provider: NVIDIA cloud inference
```

## Steps to Reproduce

1. On a DGX Spark that already hosts other sandboxes, record which ports in the documented dashboard range are in use, and confirm several are free:

```bash
for p in $(seq 18789 18799); do
ss -ltnH | awk '{print $4}' | grep -qw $p && echo "$p busy" || echo "$p free"
done
```

2. Choose a gateway port that has never been used on this host and confirm it has no state directory under the NemoClaw state root.
3. Onboard a brand-new sandbox name with the default agent and cloud inference:

```bash
NEMOCLAW_GATEWAY_PORT={unused port} nemoclaw onboard --fresh --name {new name} \
--non-interactive --yes --yes-i-accept-third-party-software
```

4. Read the tail of the output and the exit code.
5. Run `openshell forward list` and `openshell gateway list`.

## Expected Result

Onboarding allocates one of the free dashboard ports and continues.

If it genuinely cannot prove ownership of a candidate port, the error should name the port it tried, say who it believes owns it, and tell the operator what to do — the same way the port-conflict message for the managed local-inference port does, which names both the port and the holding process.

## Actual Result

```text
dashboard range 18789 free, 18790 free, 18791 busy, 18792 busy,
18793-18799 all free
gateway port chosen previously unused, no state directory
sandbox name new
onboard exit code 1

Error: Cannot allocate dashboard port: NemoClaw could not prove OpenShell forward ownership.

Onboarding did not finish. Resume from the step that failed with:
nemoclaw onboard --resume
```

No port number appears anywhere in the message. Afterwards:

```text
openshell forward list No active forwards.
openshell gateway list the gateway is registered and healthy
```

Reproduced on a second DGX Spark host with the same build, a different unused gateway port and a different new sandbox name.

## Logs

```text
onboard (plain, default agent) exit 1, "could not prove OpenShell forward ownership"
onboard (managed llama.cpp profile) exit 1, same message
free ports in the dashboard range 9 of 11
gateway state directory before run absent
openshell forward list after No active forwards.
non-Spark hosts, same build onboarding succeeded repeatedly
```

## Related issues / not duplicate of

No existing report covers this refusal.

Closest, and not duplicates:

- **#8798** — the dashboard port was baked into the sandbox image before being bound, so onboard destroyed the freshly built sandbox when the port turned out to be in use. That one is about a port that is genuinely occupied and is already fixed; here the range has free ports and no port is named at all.
- **#4084** — the dashboard bound an occupied 18789 instead of moving to the next free port in the range. That is the inverse: it bound something it should not have, where this refuses to bind anything.
- **#11963** — dashboard forward times out on macOS/WSL2. Different symptom (a timeout after the port is chosen) on different platforms.

Contributor guide

Open the contributing guide

Research direction

Start at the onboarding entry point and trace dashboard-port allocation through the OpenShell forward-ownership check. Reproduce with the documented command on a DGX Spark host, compare the port and gateway states with a non-Spark host, and inspect the outputs of `openshell forward list` and `openshell gateway list`. Done means onboarding selects a free dashboard port or reports its port, suspected owner, and an actionable next step.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux, typescript, ubuntu
Domain
cli, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.