[Linux][Onboard] portable NemoHermes onboarding always fails its final secret-boundary check with a bare PRIVILEGED_CONTROL_UNAVAILABLE token
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
Portable NemoHermes onboarding never completes, so the portable profile cannot be used
with the Hermes agent at all on this build. Every stage succeeds - the rootless
Podman runtime starts, the sandbox is created and proves CUDA usability, the Hermes
Agent gateway reports healthy, the policy presets apply, and provider-neutral
inference is verified through the sandbox runtime - and then the final secret-boundary
check emits the bare token PRIVILEGED_CONTROL_UNAVAILABLE and the run is declared
incomplete.
The token is printed on a line by itself, with no sentence explaining what privileged
control was unavailable, what was expected, or what the operator should change. The
only guidance that follows is to inspect "the validator output above", but the token
is the entire validator output.
A managed (non-portable) NemoHermes onboard on the same host, same build, same agent
and same provider completes normally and reaches Ready; the secret-boundary check does
not appear in its output at all. The only difference between the two runs is
--experimental-profile portable, which makes this specific to the portable profile.
Context, because the marker itself is not new: a change titled "preserve secret-boundary
refusals through finalization" describes this same marker appearing on fresh Portable
Hermes onboarding while the run went on to print "Deployment verified" / "Hermes is
ready" and exit 0, and states that the marker means validation could not complete rather
than that a secret was exposed. That change made the refusal terminal, which is the
correct direction. What this report adds is that the underlying validation still cannot
complete on portable Hermes, so with the refusal now enforced the profile has no working
path at all - and that the enforced failure surfaces as a bare token with no explanation.
Platform scope: Reproduced on an x86_64 Ubuntu host with an NVIDIA A100 and rootless
Podman 5.7.0; other platforms not tested this pass. Portable
NemoHermes requires GPU acceleration authority, so a GPU-less host
cannot exercise this path.
Regression: Unknown - earlier builds not tested this pass.
OpenShell issue: No - the OpenShell gateway is reachable throughout, the sandbox is
created and reaches a healthy agent gateway, and the managed profile completes against
the same gateway. The failing check belongs to NemoClaw's portable onboarding flow.
## Environment
```text
Device: x86_64 server with NVIDIA A100-SXM4-40GB
OS: Ubuntu 25.04
Architecture: x86_64
Kernel: 6.8.0-117-generic
Node.js: v22.23.2
npm: 10.9.8
Podman: podman version 5.7.0
Docker: Docker version 29.5.2, build 79eb04c
OpenShell CLI: openshell 0.0.116
NemoHermes: nemohermes v0.0.125
```
## Steps to Reproduce
```bash
Start from a host with no portable state at all, so the failure cannot be attributed
to a leftover runner or a stale auth proxy.
1. Remove any previous portable state and stop the host Ollama service, so the
portable runtime can bind its own Ollama on the loopback port:
sudo systemctl stop ollama
podman rm -f {every container}
rm -rf ~/.nemoclaw/gateways ~/.nemoclaw/portable-inference
rm -rf ~/.nemoclaw/hermes-portable-lifecycle ~/.nemoclaw/state
Confirm nothing is listening on the Ollama port and no portable runner remains.
2. Pick an unused non-default gateway port and export NEMOCLAW_GATEWAY_PORT to it.
3. Run the portable onboard:
nemohermes onboard --experimental-profile portable --fresh --name {sandbox} \
--non-interactive --yes
4. Read the tail of the output, in particular the line printed after
"Verifying provider-neutral inference through the sandbox runtime".
5. Control, on the same host and in the same session, without the portable profile:
nemohermes onboard --fresh --name {other sandbox} --non-interactive --yes
```
## Expected Result
Portable onboarding completes and the sandbox reaches Ready, the same way the managed
profile does on this host.
If the secret-boundary check genuinely cannot pass, the failure names what privileged
control was unavailable and what the operator should do about it, rather than printing
a bare token.
## Actual Result
```text
The portable run reaches the last step and then stops:
[7/8] Setting up Hermes Agent inside sandbox
Waiting for Hermes Agent gateway (up to 90s)...
Hermes Agent gateway is healthy
[8/8] Policy presets
Applied preset: personal-open-internet
Verifying provider-neutral inference through the sandbox runtime...
Provider responds through inference.local inside the sandbox
PRIVILEGED_CONTROL_UNAVAILABLE
Secret-boundary check did not complete cleanly for Hermes gateway in '{sandbox}'.
Refusing recovery; inspect the validator output above before re-running
`nemoclaw {sandbox} recover`.
Onboarding for '{sandbox}' is incomplete because a required process or
secret-boundary check did not pass.
exit 1
`doctor` disagrees with the onboarding result. Run against the same sandbox straight
after the failure it reports the lifecycle as healthy:
NemoHermes doctor: {sandbox}
[ok] Portable lifecycle: agent=Hermes; phase=active
Summary: healthy
So a sandbox whose onboarding was declared incomplete is reported healthy by the
command the failure message tells the operator to run.
The control run on the same host, without the portable profile:
nemohermes onboard --fresh --name {other sandbox} exit 0, Phase: Ready
occurrences of PRIVILEGED_CONTROL_UNAVAILABLE 0
occurrences of any "Secret-boundary" line 0
So the check that fails is only run by the portable profile, and the same host, build,
agent and provider complete successfully without it.
```
## Logs
```text
Portable run, stage by stage:
rootless Podman runtime started
sandbox creation completed, CUDA usability proven
Hermes Agent gateway healthy within 90s
policy presets personal-open-internet applied
provider-neutral inference responds through inference.local in the sandbox
secret-boundary check PRIVILEGED_CONTROL_UNAVAILABLE
onboarding exit 1, reported incomplete
Managed run, same host and session:
onboarding exit 0
status Phase: Ready
PRIVILEGED_CONTROL_UNAVAILABLE not present
"Secret-boundary" not present
```
## Related
- **#11758** — "preserve secret-boundary refusals through finalization" describes this same
marker appearing on fresh Portable Hermes onboarding while the run still printed
"Deployment verified" / "Hermes is ready" and exited 0, and notes the marker means
validation could not complete rather than that a secret was exposed. Making the refusal
terminal is the right direction; this report is that the underlying validation still
cannot complete on portable Hermes, so the profile now has no working path, and that the
enforced failure surfaces as a bare token with no explanation.
- **#10486** — Hermes Portable crashes with an uncaught exception on pre-existing gateway
state. That is about pre-existing state and an uncaught exception; this run starts from a
fully cleared host and fails a specific named check with a controlled message.
- **#10740** — onboarding refused when an empty portable config directory is not exactly
mode 0700. That is a permissions precondition rejected up front; here onboarding runs to
the last step before failing.
Reproduced three times on this host, each from a fully cleared portable state (no leftover
runner, no stale auth proxy, host Ollama stopped so the portable runtime owns its loopback
port). Two earlier attempts that failed for other reasons — a port collision with the host
Ollama service, and a stale auth proxy bound to a non-loopback address — were eliminated
before this result was recorded.
Contributor guide
Research direction
Start with the `nemohermes onboard --experimental-profile portable` entry point and trace the portable flow at the final secret-boundary check after provider-neutral inference. Reproduce from cleared portable state, compare with managed onboarding, and inspect the validator output and `doctor` result. Done means the portable check completes or reports the unavailable privileged control and an actionable next step instead of a bare token.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- cli, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100