[Inference] Sandbox inference invocation probe reports false failure while route/backend/upstream all report healthy
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
The sandbox-scoped status command (and rebuild's inference-route preflight, which shares the same probe) reports the managed inference route as unreachable/unhealthy with a "sandbox inference invocation probe" failure, while every other health signal in the same output — route reachability, upstream provider health, and Phase — reports healthy/Ready. A real inference request sent through the same route immediately succeeds. This means the probe itself is producing a false failure, not detecting a real outage.
This symptom was previously tracked as GitHub issue #10709, which was closed as not_planned/can't-reproduce. Filing fresh because the close reasoning does not hold: "can't reproduce" would only explain a genuinely broken backend, but the same route serves real traffic successfully at the same moment the probe reports failure, and multiple independent health signals (route reachability, upstream health, host-level backend checks) all report healthy at that exact moment too. A route that is verifiably working cannot be an unreproducible/external problem — the probe itself, not the backend, is the thing producing the wrong answer.
Supporting evidence gathered after the prior close:
1. On a compatible-endpoint provider, the same route served a real inference request successfully immediately after the probe reported HTTP 503, and the upstream-health check for that same run was skipped with "Endpoint URL is not known" — suggesting NemoClaw itself loses track of the endpoint URL for its own probe while still routing real traffic to it correctly.
2. Reproduced independently on two separate Ubuntu x86_64 hosts with a local Ollama provider on the current release: one via a plain `status` call showing every other signal healthy, the other via `rebuild --yes` failing its inference preflight on the strength of the probe alone, reproduced identically on immediate retry, with the host's own Ollama backend independently confirmed healthy outside NemoClaw (`curl 127.0.0.1:11434/api/tags`, `systemctl is-active ollama`).
Platform scope: Reproduced across multiple platforms and providers — DGX Spark, Jetson Thor, Ubuntu 26.04 GPU, and WSL2 x86 (original report); Ubuntu 24.04 x86_64 with a compatible-endpoint provider; two independent Ubuntu x86_64 hosts with a local Ollama provider (this report). Not platform- or provider-specific.
Regression: Unknown — earlier versions not checked with this exact contract.
## Environment
```text
Device: Multiple (see Platform scope)
OS: Ubuntu x86_64 (this report's direct evidence)
NemoClaw: v0.0.119 (this report's direct evidence); also reproduced on v0.0.116
```
## Steps to Reproduce
1. Onboard an OpenClaw sandbox with a working inference provider (local Ollama or a compatible-endpoint provider) and reach Phase Ready.
2. Run: `nemoclaw status`
3. Separately, run: `nemoclaw rebuild --yes`
4. Independently confirm the inference backend/route is actually healthy outside NemoClaw (direct curl to the provider, or a real in-sandbox inference request).
## Expected Result
Status and rebuild's inference preflight agree with the sandbox's actual, independently-confirmed inference health. A route that is genuinely serving requests should not be reported as failing an invocation probe.
## Actual Result
`status`:
```text
Inference: unreachable ... sandbox inference invocation probe was unavailable
```
(Inference route reachability, Ollama backend, and auth proxy all report reachable/healthy; `nemoclaw doctor` reports the sandbox healthy with exit 0.)
`rebuild --yes`:
```text
Rebuild preflight failed: recorded inference credentials or route were rejected: sandbox inference invocation probe was unavailable
```
(Host Ollama backend independently confirmed healthy via curl and systemctl; reproduced identically on immediate retry.)
Prior evidence (compatible-endpoint provider, different host):
```text
Inference: unhealthy (https://inference.local/v1/models)
Inference gateway did not serve an inference request: sandbox inference invocation probe returned HTTP 503.
Inference (route reachability): reachable (https://inference.local/v1/models)
Inference (upstream): not probed (Endpoint URL is not known; skipping reachability check.)
Phase: Ready
status exit code: 1
```
A real inference request over the same route immediately afterward succeeded.
## Logs
```text
Inference: unreachable ... sandbox inference invocation probe was unavailable
Rebuild preflight failed: recorded inference credentials or route were rejected: sandbox inference invocation probe was unavailable
```
## Related
Related to #10709 (closed as not_planned/can't-reproduce) — filed as a fresh issue because evidence gathered after that close contradicts the close reasoning; see Description.
Contributor guide
Assessment
This issue has not been assessed yet.