NVIDIA / NVIDIA/NemoClaw

[DGX Spark][Onboard] managed llama.cpp doctor and destroy exit 1 with "endpoint does not match persisted authority" after interactive onboard

Open
#10,220 5 comments 0 reactions 1 assignee Claimed by @yanyunl1991 View on GitHub
area: local-models area: onboarding area: sandbox NV QA platform: dgx-spark
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

> [!NOTE]
> **Re-verified reopening of #9584** — closed 2026-08-20 via #9666 ("fix(onboard): stabilize Docker authority across terminals"), with a related prerequisite fix in #9636 (fixes #9599) and a sibling non-interactive-path fix in #9953 (fixes #9888). All three are confirmed ancestors of the build tested below. Despite that, a genuine interactive-terminal onboard still hits the exact original symptom:
> - v0.0.113: `destroy` still failed with the same error (verification held at the time, pending #9888).
> - v0.0.114: a fresh, uninterrupted interactive onboard (real TTY, no `--fresh`, no interrupt) still fails identically on both `doctor` and `destroy` immediately after onboarding completes.
>
> Opening as a new issue (instead of reopening #9584) so this re-enters triage visibly.

## Description

After completing a managed llama.cpp onboard via the interactive menu on DGX Spark, `nemoclaw doctor` exits 1 with "Qualified container endpoint does not match persisted authority". `nemoclaw destroy` also exits 1 with the same error. The non-interactive path (`NEMOCLAW_PROVIDER=install-llama-cpp`) does NOT reproduce this — `doctor` exits 0 healthy on that path.

Platform scope: Reproduced on DGX Spark only; other platforms not tested (managed llama.cpp is only supported on DGX Spark).
Regression: previously believed fixed via #9666; re-verification on v0.0.113 and v0.0.114 shows the original symptom unchanged.
OpenShell issue: No.

## Environment

```text
Device: DGX Spark (GB10, aarch64)
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Node.js: v22.23.1
npm: 10.9.8
Docker: 29.2.1
OpenShell CLI: 0.0.101
NemoClaw: v0.0.110 (originally filed on); re-verified still failing on v0.0.113 and v0.0.114
OpenClaw: v2026.7.1
```

## Steps to Reproduce

1. On DGX Spark, run: `NEMOCLAW_EXPERIMENTAL=1 nemoclaw onboard --name llama-cpp-local`
2. At provider selection, choose "Managed llama.cpp: Meta Muse Glimmer 30B on one DGX Spark (recommended)"
3. Allow onboard to complete (sandbox reaches Ready state)
4. Run: `nemoclaw llama-cpp-local status` (confirms Ready, provider=llama-cpp-local)
5. Run: `nemoclaw llama-cpp-local doctor`
6. Observe: exits 1 with "[fail] Managed llama.cpp runtime: conflict: Qualified container endpoint does not match persisted authority"
7. Run: `nemoclaw llama-cpp-local destroy --yes`
8. Observe: exits 1 with the same endpoint mismatch error

Workaround: the non-interactive path is unaffected:

```bash
NEMOCLAW_EXPERIMENTAL=1 NEMOCLAW_PROVIDER=install-llama-cpp NEMOCLAW_YES=1 \
nemoclaw onboard --non-interactive --name llama-managed
nemoclaw llama-managed doctor # exits 0 healthy
```

## Expected Result

`nemoclaw llama-cpp-local doctor` exits 0 with all checks healthy.
`nemoclaw llama-cpp-local destroy --yes` exits 0 and removes the sandbox cleanly.

## Actual Result

```text
nemoclaw llama-cpp-local doctor
[fail] Managed llama.cpp runtime: conflict: Qualified container endpoint does not match persisted authority
Exit code: 1

nemoclaw llama-cpp-local destroy --yes
Error: Qualified container endpoint does not match persisted authority
Exit code: 1
```

## v0.0.114 re-verification (fresh interactive repro on this build)

```text
nemoclaw doctor immediately after a clean, uninterrupted interactive onboard:
[fail] Managed llama.cpp runtime: conflict: Qualified container endpoint does not match persisted authority.; endpoint https://inference.local/v1
hint: re-run `nemoclaw onboard` for '' to recover the exact runtime
Summary: attention needed (1 failed, 0 warning(s))
EXITCODE:1

nemoclaw destroy --yes (and separately --force):
Refusing to destroy sandbox '': Managed llama.cpp cleanup preflight failed: Qualified container endpoint does not match persisted authority.
The sandbox was not deleted. Resolve the reported cleanup preflight failure and retry destroy.
EXITCODE:1
```

## Logs

```text
--- nemoclaw llama-cpp-local doctor ---
[fail] Managed llama.cpp runtime: conflict: Qualified container endpoint does not match persisted authority.
Summary: attention needed (1 failed, 0 warning(s))
Exit code: 1

--- nemoclaw llama-cpp-local destroy --yes ---
Error: Qualified container endpoint does not match persisted authority.
Exit code: 1

--- receipt.json engineAuthority at failure ---
authorityId: docker:a70733d50dfb07970f4a024e6914d84cdff86b0f2537fb0cb7722010e84f9851
endpoint: host.openshell.internal:8081
runtime.name: nemoclaw-llama-cpp
runtimeId: 1660f44399f68255d1a4b02562cf50312d80f5be54fd64bdfd200ae72bcb48f8

Persisted authority does not match container when created via interactive menu.
Non-interactive path (NEMOCLAW_PROVIDER=install-llama-cpp) doctor exits 0 healthy.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.