NVIDIA / NVIDIA/NemoClaw

[DGX Station][Install] Station host-preparation failure ("Unqualified failed system units") exits 0 instead of nonzero

Open
#11,511 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

**Description**

When `nemoclaw.sh --force-station-install` (Station validation-only path) triggers `ensure_station_express_host` -> `run_station_host_preparation`, and that preparation fails for a reason not covered by the 0/10/11/12 status cases (in this repro: pre-existing failed systemd units unrelated to NemoClaw -- cloud-init.service, fluent-bit.service, fwupd-refresh.service, several sssd-*.socket units -- causing prepare-dgx-station-host.sh's own "Unqualified failed system units block Station preparation" check to fail), install.sh prints `[ERROR] DGX Station host preparation failed. Review the station-bootstrap log above, correct the reported host state, and rerun the installer.` via the standard `error()` helper (which calls `error_with_status 1 ...`, i.e. `exit 1`) -- but the actual process exit code observed by the caller is 0, not 1.

This was reproduced in a real interactive terminal (tmux, genuine PTY allocated, not a piped/non-TTY artifact) with a single semicolon-chained command so there is no ambiguity about which process's exit status was captured. `set -euo pipefail` is active at the top of install.sh and `error()` unconditionally calls `exit 1`, so the direct call chain (`prepare_installer_host` -> `ensure_station_express_host`, no subshell/command-substitution wrapping at either call site) should propagate a nonzero exit to the shell -- something in the actual execution path is not doing so. Root cause not fully isolated (did not have time to bisect further within this session), but the discrepancy itself is clean and repeatable.

Platform scope: Reproduced on DGX Station GB300 (aarch64) only; the failing code path (`ensure_station_express_host`, `prepare-dgx-station-host.sh`) is Station-specific by design, so this is not expected to reproduce on other platforms.
Regression: Unknown - earlier versions not tested this session.
OpenShell issue: No.

**Environment**

Device: DGX Station GB300 (DGX_SWBUILD_VERSION 7.6.0, no-OTA)
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
NemoClaw: v0.0.122 (NEMOCLAW_INSTALL_TAG pinned explicitly)

**Steps to Reproduce**

1. On a DGX Station GB300 host that has one or more pre-existing failed systemd units unrelated to NemoClaw (this repro had cloud-init.service, fluent-bit.service, fwupd-refresh.service, sssd-autofs.socket, sssd-nss.socket, sssd-pam-priv.socket, sssd-pam.socket in `failed` state -- check with `systemctl --failed`), run in a real interactive terminal:
`NEMOCLAW_INSTALL_TAG=v0.0.122 bash nemoclaw.sh --force-station-install`
2. Answer `Y` at the "Run validation-only Station checks with these settings?" prompt.
3. Let it run to completion; immediately after, in the same shell, run: `echo EXIT_WAS=$?`

**Expected Result**

The command exits nonzero (1, matching the `error()` helper's documented behavior) when Station host preparation fails, so a script/CI wrapper checking the exit code correctly detects the failure.

**Actual Result**

[station-prepare] ERROR: Unqualified failed system units block Station preparation
[ERROR] DGX Station host preparation failed. Review the station-bootstrap log above, correct the reported host state, and rerun the installer.
EXIT_WAS=0

The error text is correct and informative, but the process exit code is 0 -- indistinguishable from success to any automated caller.

**Logs**

Full captured tmux pane transcript (single continuous session, real PTY):
[INFO] Detected DGX Station.
[INFO] Using validation-only Station checks.
[WARN] Proceeding with explicit --force-station-install intent; only DGX release metadata qualification is bypassed. Station GB300 hardware, workload quiescence, and factory-runtime health checks remain required.
[INFO] Validating the existing Station GPU and local container runtime. Host packages, the NVIDIA driver, and runtime configuration will not be changed.
[INFO] DGX Station host preparation log: /localhome/local-lynnh/station-bootstrap-logs/station-prepare-apply-20260911T031119Z.log
[WARN] DGX release metadata allowlist bypassed by explicit --force-station-install intent; all hardware and factory-runtime health checks remain required
[WARN] unqualified failed unit: cloud-init.service
[WARN] unqualified failed unit: fluent-bit.service
[WARN] unqualified failed unit: fwupd-refresh.service
[WARN] unqualified failed unit: sssd-autofs.socket
[WARN] unqualified failed unit: sssd-nss.socket
[WARN] unqualified failed unit: sssd-pam-priv.socket
[WARN] unqualified failed unit: sssd-pam.socket
[station-prepare] 2026-09-11T03:11:20Z ERROR: Unqualified failed system units block Station preparation
[ERROR] DGX Station host preparation failed. Review the station-bootstrap log above, correct the reported host state, and rerun the installer.
REAL_EXIT_DONE=0

Contributor guide

Open the contributing guide

Research direction

Start with nemoclaw.sh and install.sh, tracing prepare_installer_host through ensure_station_express_host and run_station_host_preparation into prepare-dgx-station-host.sh. Reproduce the failure on a DGX Station with failed systemd units and inspect how the error path returns its status. Done means the same host-preparation failure produces a nonzero process exit while preserving the existing error message.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, shell
Domain
cli, devops, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.