[Linux][Sandbox] start treats a transient inference.local HTTP 503 as a final recovery failure
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Investigation Summary
- NemoClaw v0.0.126 reproduces the failure on a clean Ubuntu 22.04 Docker host with Hermes.
- `stop` then `start` preserves the container identity and returns the sandbox to a running state.
- `start` exits 1 after its one inference request receives HTTP 503.
- A status-only request one millisecond after `start` exits also receives HTTP 503.
- The same route later returns HTTP 200 without a restart, repair, credential change, or route change.
## Description
`nemohermes start` treats one HTTP 503 inference response as a final startup failure.
The container remains running with the same identity. The route later serves HTTP 200 without another lifecycle action. This response is a transient provider or gateway availability result, not evidence that startup permanently failed.
NemoClaw already defines HTTP 429, 502, 503, and 504 as retriable inference probe statuses. Sandbox status uses that shared policy. The start path calls `probeSandboxInferenceInvocation` once and does not apply the policy.
This is distinct from #11790. PR #11797 fixes the stopped-phase lifecycle transition. The reproduced v0.0.126 run confirms that `connect --probe-only` now restores the sandbox to Ready with the same container identity.
Issue #10709 records the product decision that ordinary status does not retry, but NemoClaw can retry when it owns a settling transition, including managed-gateway recovery. `start` owns that transition.
Expected behavior: after a stopped sandbox restarts, `start` retries only the shared transient inference statuses within a fixed bound. It must still fail immediately for authentication, model, malformed-response, transport, and non-retriable server failures.
## Reproduction Steps
1. Install NemoClaw v0.0.126 at commit `ef7299010fa731d4075a29363aa60ee3c48f2eaf`.
2. Onboard Hermes with NVIDIA Endpoints and model `nvidia/nemotron-3-super-120b-a12b`.
3. Confirm one successful request through `https://inference.local/v1/chat/completions`.
4. Run `nemohermes stop`.
5. Run `nemohermes start`.
6. Record the exit code, container identity, and the status of the first bounded inference request.
## Environment
- Host: Brev `g6e.xlarge`, one NVIDIA L40S
- OS: Ubuntu 22.04, x86_64
- Runtime: Docker 29.8.0
- OpenShell: 0.0.116
- NemoClaw: v0.0.126
- Candidate commit: `ef7299010fa731d4075a29363aa60ee3c48f2eaf`
- Agent: Hermes 0.20.6
- Provider: NVIDIA Endpoints
## Debug Output
The diagnostic did not retain a response body or credential. It recorded only exit codes, timing, HTTP status, and identity equality:
```json
{
"operation": "diagnostic-explicit-start-first-http-status",
"stopExitCode": 0,
"startExitCode": 1,
"diagnosticExitCode": 0,
"httpStatus": "503",
"stopDurationMs": 1730,
"startDurationMs": 38504,
"startToDiagnosticMs": 1,
"diagnosticDurationMs": 236,
"containerCount": 1,
"sameIdentity": true,
"containerRunning": true
}
```
The earlier v0.0.126 lifecycle checks also produced these results:
```text
stop -> explicit start 41.768 s, same container, Ready
stop -> connect --probe-only 40.810 s, same container, Ready
post-recovery request HTTP error
later diagnostic request HTTP 200, no repair between requests
```
## Scope Boundary
The first fix should change only `start` inference validation:
- Reuse `isTransientInferenceInvocationFailure` and the existing retry helper.
- Retry only HTTP 429, 502, 503, and 504.
- Keep a fixed attempt count and delay.
- Keep the first request and each retry inside the existing lifecycle fence.
- Preserve immediate failure for HTTP 401, 403, 404, 500, invalid response bodies, unavailable probes, and transport failures.
- Add deterministic tests for transient success, transient exhaustion, and non-retriable failure.
Do not broaden this issue to ordinary `status`. Issue #10709 records that status must report its current observation without retry.
## Checklist
- [x] I confirmed this bug is reproducible.
- [x] I searched existing issues and this is not a duplicate.
Contributor guide
Assessment
This issue has not been assessed yet.