NVIDIA / NVIDIA/NemoClaw

[DGX Station][Onboard] After an upgrade replaces the gateway binary, preflight rejects NemoClaw's own healthy gateway as an ambiguous port owner and blocks all onboarding

Open
#10,460 0 comments 0 reactions 0 assignees View on GitHub
area: install area: onboarding area: sandbox platform: dgx-station
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

An upgrade replaces `/home//.local/bin/openshell-gateway` on disk but does not restart the
`nemoclaw-openshell-gateway` user service. The already-running gateway keeps serving happily from the now-deleted
inode. From that moment every `nemoclaw onboard` fails at preflight:

```
✗ The gateway port is held by an incompatible or ambiguous owner.
✗ System readiness could not confirm required capabilities: gateway.version.compatible.
Gateway port 8080 is occupied by openshell-gatew (PID NNNN). Confirm PID NNNN is not another NemoClaw
gateway. ... Stop that process through its service manager, or signal only the matching PID ...
```

The process it is refusing is NemoClaw's own gateway: it is the systemd user service's MainPID, the service is
active, `openshell gateway info` reports Status healthy and Version 0.0.106, and `openshell --version` is the
same 0.0.106. Nothing is ambiguous and nothing is version-incompatible — the only thing wrong is that the
executable behind the running process no longer exists at its path.

The remediation text points the user at `sudo lsof` and at killing a PID, which is both alarming and the wrong
action. The actual fix is one command: `systemctl --user restart nemoclaw-openshell-gateway`.

Impact: onboarding is blocked on every host that has been upgraded and not rebooted or manually restarted —
which is the normal state right after an in-place upgrade.

Platform scope: Observed on DGX Station GB300 (aarch64, Ubuntu 24.04) with NemoClaw v0.0.114 / OpenShell 0.0.106.
The same signature was seen on a DGX Spark during v0.0.114 upgrade testing the previous day.
Regression: Unknown — not tested on earlier releases.

## Environment

```
Device: DGX Station GB300 (gb300ws-sku2-or-09)
OS: Ubuntu 24.04 LTS
Architecture: aarch64
GPU: NVIDIA GB300, 256703 MiB
Docker: 29.2.1
NemoClaw: v0.0.114
OpenShell CLI: 0.0.106
```

## Steps to Reproduce

```
1. On a host where NemoClaw has been upgraded in place and the gateway user service has been running since
before the upgrade, confirm the gateway is healthy and version-matched:
systemctl --user is-active nemoclaw-openshell-gateway
systemctl --user show nemoclaw-openshell-gateway -p MainPID --value
openshell gateway info
openshell --version
2. Confirm the running gateway's executable was replaced on disk:
readlink -f /proc/$(systemctl --user show nemoclaw-openshell-gateway -p MainPID --value)/exe
-> the path ends in "(deleted)"
3. Run any onboarding:
NEMOCLAW_YES=1 nemoclaw onboard --non-interactive --name probe
4. Restart the gateway service and repeat step 2 and step 3:
systemctl --user restart nemoclaw-openshell-gateway
```

## Expected Result

Preflight recognises a healthy, version-matched gateway that belongs to NemoClaw and owns the port, and lets
onboarding continue — the same "Port 8080 already owned by healthy NemoClaw runtime" result it prints once the
service has been restarted. If the running gateway genuinely cannot be validated, the guidance names the real
remedy (restart the gateway service) rather than sending the user to `lsof` and `kill`.

## Actual Result

Step 1 — the gateway is healthy and version-matched:

```
systemctl --user is-active -> active
MainPID ->
openshell gateway info -> Gateway: nemoclaw / Server: https://127.0.0.1:8080 / Status: healthy / Version: 0.0.106
openshell --version -> openshell 0.0.106
ss -ltnp | grep :8080 -> users:(("openshell-gatew",pid=,fd=15))
```

Step 2 — its executable is gone:

```
readlink -f /proc//exe
-> /home//.local/bin/openshell-gateway (deleted)
```

Step 3 — onboarding is refused:

```
[1/8] Preflight checks
✗ The gateway port is held by an incompatible or ambiguous owner.
✗ System readiness could not confirm required capabilities: gateway.version.compatible.
Gateway port 8080 is occupied by openshell-gatew (PID ). Confirm PID is not another NemoClaw
gateway. Recheck the listener set immediately before stopping a process: sudo lsof -i :8080 -sTCP:LISTEN -P -n.
Stop that process through its service manager, or signal only the matching PID from that fresh result before
retrying.
Onboarding did not finish.
```

Step 4 — restarting the service is the whole fix:

```
systemctl --user restart nemoclaw-openshell-gateway -> exit 0, new MainPID
readlink -f /proc//exe -> /home//.local/bin/openshell-gateway (no "(deleted)")

[1/8] Preflight checks
...
✓ Port 8080 already owned by healthy NemoClaw runtime (OpenShell gateway)
✓ NVIDIA GPU detected (NVIDIA GB300, 256703 MB)
```

Same port, same owning role, same versions before and after. The only variable is whether the running process's
binary still exists on disk.

## Logs

```
# the unit that owns the port
ExecStartPre=/home///.local/bin/openshell-gateway generate-certs --output-dir ${OPENSHELL_LOCAL_TLS_DIR} --server-san host.openshell.internal
ExecStart=/home///.local/bin/openshell-gateway

# note the doubled slash in the unit's ExecStart path; the running cmdline carries it too
/proc//cmdline -> /home///.local/bin/openshell-gateway
It is not clear whether the ownership check compares paths textually; if it does, the doubled slash may be a
second reason the owner is classified as foreign. Restarting the service fixes the symptom either way.

# workaround for anyone blocked
systemctl --user restart nemoclaw-openshell-gateway
```

Contributor guide

Open the contributing guide

Research direction

Start with the onboarding preflight check that validates ownership of gateway port 8080, then reproduce the upgrade state using the listed systemctl, openshell, and /proc commands. Compare the running service MainPID, executable path, health, and version before and after restarting nemoclaw-openshell-gateway. Done means a healthy, version-matched NemoClaw gateway with a deleted executable is handled correctly, with accurate restart guidance when validation is impossible.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, typescript
Domain
cli, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.