NVIDIA / NVIDIA/NemoClaw

[Hermes/docker] Host-local llama.cpp per documented 127.0.0.1:8081 binding is unreachable from the sandbox inference route; onboarding completes without detecting it

Open
#11,626 0 comments 0 reactions 1 assignee Claimed by @rluo8 View on GitHub
needs: triage
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

### Investigation Summary

- Onboarding a Hermes sandbox against a host-local llama.cpp bound exactly as
documented (-p 127.0.0.1:8081:8081) completes with the inference step marked
complete, but all agent inference fails.
- Every in-sandbox inference attempt hangs ~20-30s then fails: sandbox proxy log
shows "routing proxy inference request" followed by OCSF NET:FAIL; the gateway
journal shows no inference request ever arriving.
- While attempts are in flight, the sandbox openshell-sandbox process shows
SYN-SENT retries to 172.18.0.1:8081 (the Docker bridge IP, i.e.
host.openshell.internal) — a SYN blackhole; curl from the sandbox to that
address times out with no RST.
- Recreating the llama container with a dual binding
(-p 127.0.0.1:8081:8081 -p 172.18.0.1:8081:8081) made both sandboxes report
"Inference: healthy" / "route reachability: reachable" immediately, and real
completions flowed through https://inference.local/v1 — confirming the
sandbox-facing route dials the bridge IP, where nothing was bound.
- Unknown: whether the route is intended to dial the host from the sandbox at
all, or should relay through the host-side gateway (which would make the
documented 127.0.0.1:8081 binding work as-is).

### Description

With the llama.cpp "existing server" configured exactly as documented (server
listening on 127.0.0.1:8081, Docker port binding -p 127.0.0.1:8081:8081),
nemohermes onboard succeeds and reports the inference route healthy. The
sandbox is in fact unable to reach the local provider: the sandbox-facing
route targets host.openshell.internal:8081 (Docker bridge IP), which stock
Docker iptables blackholes for a port only published on 127.0.0.1, so every
inference attempt hangs ~20-30s and fails (OCSF NET:FAIL).

Expected: the documented 127.0.0.1:8081 binding works for sandbox inference,
or at minimum onboarding detects the broken sandbox-to-upstream hop instead of
reporting the route healthy on host-side reachability alone
(src/lib/actions/inference-set.ts intentionally skips gateway-side verify for
local providers: "host-side verification cannot resolve the container hostname,
so it is skipped" — and no in-sandbox probe is run either).

Workaround that restores inference: publish the llama port on the bridge IP as
well (-p 172.18.0.1:8081:8081 in addition to the 127.0.0.1 binding).

Related (not duplicates): #10543 (onboard claims healthy route that status
reports unhealthy — same readiness-disagreement class, different mechanism),
#8219 (stale resolv.conf after host network change), #11444 (host-process
llama.cpp bridge probe on WSL).

### Reproduction Steps

1. Run a llama.cpp server with Docker binding -p 127.0.0.1:8081:8081
(the documented existing-server binding), API key enabled.
2. Run nemohermes onboard selecting Local llama.cpp. Onboarding completes and
reports the inference route healthy.
3. Trigger any inference from the agent (e.g. a chat turn). It hangs ~20-30s
and fails; nemohermes status reports
"Inference: unreachable (https://inference.local/v1/models)".
4. (Confirming the hop) Recreate the llama container with the dual binding
-p 127.0.0.1:8081:8081 -p 172.18.0.1:8081:8081. Inference immediately
works; status reports "Inference: healthy".

### Environment

- OS: Fedora Linux 44 (Server Edition), kernel 7.1.13-200.fc44.x86_64, x86_64
- GPU: NVIDIA GeForce RTX 4090 (driver 610.57.04), Docker CDI GPU passthrough
- Docker: Docker Engine 29.7.2 (docker driver)
- NemoClaw / NemoHermes: v0.0.109 -> v0.0.123 (upgraded 2026-09-12)
- OpenShell: 0.0.101 -> 0.0.106 (upgraded by the installer during upgrade-sandboxes)
- Hermes agent runtime: 0.19.0 -> 0.20.6 (sandbox image rebuild)
- Node.js: v24.18.0
- Layout: 2 sandboxes (leryl-01, frankiebox-02) sharing one gateway "nemoclaw" on default port 8080 and one host-local llama.cpp server (llama-cpp-local, ornith:9b, 127.0.0.1:8081); each sandbox has a telegram channel.

### Debug Output

[nemoclaw-debug-postfix.tar.gz](https://github.com/user-attachments/files/32147035/nemoclaw-debug-postfix.tar.gz)
[nemoclaw-debug.tar.gz](https://github.com/user-attachments/files/32147036/nemoclaw-debug.tar.gz)

Note: the bundle captures the state after the workaround was applied (inference
healthy); the pre-fix failure evidence is in the Logs field below.

### Logs

```shell
Pre-fix, sandbox proxy log (docker logs ):

2026-09-12T01:47:35.819Z OCSF NET:OPEN [INFO] ALLOWED inference.local:443
2026-09-12T01:47:35.819Z INFO openshell_router: routing proxy inference request
2026-09-12T01:47:55.721Z OCSF NET:FAIL [LOW] inference.local:443

While an attempt was in flight, sandbox main netns (ss -tnpo):

SYN-SENT 0 1 172.18.0.4:42374 172.18.0.1:8081 users:(("openshell-sandb",pid=1,fd=27)) timer:(on,117ms,4)

Pre-fix, from inside the sandbox:

curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://172.18.0.1:8081/health
# -> 000 (connection timeout; no RST — SYN dropped by the Docker DOCKER chain)

Pre-fix, nemohermes status:

Inference: unreachable (https://inference.local/v1/models)
Inference gateway unreachable on https://inference.local/v1/models from inside the sandbox.

Post-fix (dual binding):

curl ... http://172.18.0.1:8081/health -> http=200
Inference: healthy (https://inference.local/v1/models)
Inference (route reachability): reachable (https://inference.local/v1/models)
real completion through the sandbox proxy path -> http=200, model responded
```

### Checklist

- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.