`status` inference health can only attest a fresh exec's environment, never the serving process's — surface process-level (self-reported) health
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Problem Statement
This is Stage B of the scope split proposed in #6846 ([comment](https://github.com/NVIDIA/NemoClaw/issues/6846#issuecomment-4978877386)): the structurally distinct gap @mikemason identified there, filed separately so it doesn't block the Stage A probe-vocabulary work.
Every inference health signal `status`/`doctor` can produce today is measured from a **freshly spawned `sandbox exec`** (`buildSandboxInferenceRouteProbeArgs`, `src/lib/actions/sandbox/connect-inference-route-probe.ts:82`, returns `["sandbox", "exec", ...]`). The file itself documents why that exec is trustworthy: "OpenShell injects the per-sandbox trust bundle into each exec process" (`connect-inference-route-probe.ts:23`). That injection is exactly what makes the probe blind to the process it stands in for: the **long-running agent gateway serving real traffic** has whatever environment it was *started* with, and no probe running in a fresh exec can attest it.
The environment contract at stake is not just the CA bundle. On a live openclaw sandbox the serving process depends on, among others: `NODE_EXTRA_CA_CERTS`, `SSL_CERT_FILE`/`CURL_CA_BUNDLE`, `HTTPS_PROXY`/`NODE_USE_ENV_PROXY` (all egress goes through the in-sandbox proxy), and `NODE_OPTIONS` preloads. In the reproduction below, a serving process without that contract cannot even *resolve* `inference.local` — while every status leg stays green.
@mikemason hit this organically (#6846 discussion): after working around #6635 by hand-starting the gateway, `NODE_EXTRA_CA_CERTS` was missing, every real model call failed with `SELF_SIGNED_CERT_IN_CHAIN`, and `status` reported both inference legs healthy throughout. The reproduction below confirms the same class of failure on current `main` with a local provider — no corporate proxy or TLS interception involved — by emulating that recovery scenario.
Stage A (#6846: authenticated invocation verdicts) shrinks the reachable-vs-invocable gap but inherits this limit by construction: an authenticated invocation *in a fresh exec* is still an assertion about the fresh exec.
## Reproduction (by execution, `main` @ `c1bda8069`, x86_64/WSL2, docker driver, ollama-local)
**1. Healthy baseline.** Fresh `onboard --name stageb-demo --agent openclaw` — all green, and the serving process carries the same injected trust/proxy contract as a fresh exec (the only difference in the compared env sets is the order of the `NODE_OPTIONS` preloads).
**2. Enter the #6635 state, then recover by hand.** Killed the in-container supervision loop (`nemoclaw-start`) and the gateway, then restarted the gateway the way a stranded user does after #6635 — by hand, without the injected contract (`env -i HOME=... PATH=... TERM=... openclaw gateway run`, emulating the environment loss @mikemason hit for real). The gateway comes up and serves:
```
$ openshell -g nemoclaw-18899 sandbox exec --name stageb-demo -- sh -c 'curl -s --max-time 5 -o /dev/null -w "gateway: HTTP %{http_code}\n" http://127.0.0.1:18791/'
gateway: HTTP 200
```
Its environment, read from `/proc//environ`, is now three variables (`HOME`, `PATH`, `TERM`) — no CA bundle, no proxy, no `NODE_OPTIONS` preloads.
**3. `status` reports healthy inference on every leg:**
```
Sandbox: stageb-demo
Model: llama3.2:1b
Provider: ollama-local
Inference: healthy (https://inference.local/v1/models)
Inference (ollama backend): healthy (http://127.0.0.1:11434/api/tags)
Inference (auth proxy): healthy (http://127.0.0.1:11435/api/tags)
```
**4. The serving process cannot make the call `status` just vouched for.** Same sandbox, same instant, same URL — only the environment differs:
```
[A] serving-process env:
curl: (6) Could not resolve host: inference.local
HTTP 000
[B] fresh-exec env:
HTTP 200
```
(`[A]` replays the call under the hand-started process's exact environment; `[B]` is the same call in the fresh exec's injected environment. `inference.local` only resolves/routes through the injected egress-proxy config, so the broken serving process fails at name resolution before TLS is even attempted — an even earlier failure than the `SELF_SIGNED_CERT_IN_CHAIN` variant in #6846.)
The window is not exotic: any path where the serving process's env diverges from the current exec-injection contract can produce it — manual recovery after #6635, an env/CA change after the process started, or any supervisor-less restart. And it is silent by construction: the probes get a *fresh, correct* environment every time, so they can never see it.
## Desired Behavior
- `status`/`doctor` gain a leg that reports what only the serving process can know, sourced from the process itself rather than from a fresh probe, e.g.:
- `Serving process (openclaw gateway): last model call OK (2m ago)`
- `Serving process (openclaw gateway): failing since 06:54 (name resolution)`
- Where no self-report is available, say so explicitly instead of staying silently green — the honesty pattern #6743 and #6887 already applied elsewhere in status (`Runtime health: not checked`).
- Rendering keeps the two kinds of facts distinguishable: "a fresh exec can invoke the model" (Stage A) vs "the process serving traffic can" (this issue).
## Constraints and Non-goals
- Complements Stage A (#6846); neither replaces the other. Non-goal: changing probe semantics — that's Stage A's scope.
- The self-report should be **agent-pluggable**, not OpenClaw-specific. The manifest already declares a per-agent health surface (`health_probe`, `agents/openclaw/manifest.yaml:27`), currently used by onboard/recovery as a liveness gate but never surfaced by `status` as serving-process health. Agents with weaker internals (hermes) degrade to `not checked`.
- Non-goal: NemoClaw running its own monitoring daemon. The point is to *read* what the serving process already knows.
## Implementation Idea (non-binding)
The primitive already exists on the agent side. The hand-started (broken) gateway in the repro logs, verbatim:
```
2026-07-16T06:54:17.071+00:00 [health-monitor] started (interval: 300s, startup-grace: 60s, channel-connect-grace: 120s)
```
i.e. OpenClaw runs an internal health monitor on a 5-minute cycle (plus a heartbeat, `intervalMs: 1800000`). The 300s interval lines up with the "OpenClaw does an LLM ping every 5 minutes" behavior @mikemason described in #6846 (I observed the monitor running, not what it checks — that part is his report). A first cut could be:
- extend the agent manifest contract (`health_probe` → e.g. a `self_report` entry) declaring where/how the serving process exposes its own last health verdict;
- have `status`/`doctor` fetch and render that verdict with its age, replacing silence with `not checked` when the agent declares nothing.
@mikemason mentioned in #6846 he may want to work on this — the design instinct (pluggable health) is his; happy to support, review, or pair on it, and Stage A on my side won't collide with it.
Contributor guide
Research direction
Start with src/lib/actions/sandbox/connect-inference-route-probe.ts, especially the probe builder at line 82 and its environment note near line 23, then inspect the health_probe contract in agents/openclaw/manifest.yaml:27 and the status/doctor entry points. Done means serving-process health is reported separately from fresh-exec inference, with age or failure details, and agents without a self-report explicitly show “not checked”.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend-api-design, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100