[Ubuntu 26.04][Agent&Skills] hermes chat stalls after host network change (stale sandbox DNS)
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
After the host machine's network changes (e.g. laptop switched from home Wi-Fi to office network) without restarting or reconnecting the sandbox container, the Hermes CLI's built-in "3 retries with backoff" wraps each upstream call in a fresh DNS lookup. When the container's `/etc/resolv.conf` still points at the pre-switch resolvers (Docker snapshots `resolv.conf` at container creation and does not refresh it when the host's DNS changes), every retry stalls on DNS timeouts. A single `hermes chat --query` grows from a few seconds to ~67 s, and a typical multi-turn interactive session (~15-20 tool-using turns) compounds to ~20 min end-to-end. The original user report showed the same behaviour independent of the configured provider — both a cloud provider (`api.deepseek.com`) and a locally-deployed Ollama model exhibited the slowdown, confirming the delay is in the shared retry+DNS layer rather than any single provider.
**Platform scope:** Reproduced on Ubuntu 26.04 (x86_64) only; other platforms not tested.
**Regression:** Unknown — earlier versions not tested.
## Environment
```text
Device: Ubuntu 26.04 LTS server (x86_64)
OS: Ubuntu 26.04 LTS (kernel 7.0.0-27-generic)
Architecture: x86_64
Node.js: v22.23.1
npm: 10.9.8
Docker: 29.6.1
OpenShell CLI: 0.0.72
NemoClaw: v0.0.101
OpenClaw: Hermes agent v0.17.0 (2026.6.19)
```
## Steps to Reproduce
1. On a healthy host with working DNS, onboard a Hermes sandbox (following https://docs.nvidia.com/nemoclaw/latest/get-started/quickstart-hermes.html) and confirm `hermes chat` responds within a few seconds.
2. Simulate a network change without restarting the sandbox container — the simplest deterministic proxy for the real "home to office" transition is to point the container's `/etc/resolv.conf` at an unreachable nameserver:
```bash
docker exec -u root {container id} sh -c \
'cp /etc/resolv.conf /tmp/resolv.bak && \
printf "nameserver 203.0.113.10\noptions timeout:2 attempts:2 ndots:0\n" > /etc/resolv.conf'
```
3. Measure a single `hermes chat` inside the sandbox:
```bash
time docker exec -u sandbox {container id} sh -c \
'cd /sandbox && HOME=/sandbox hermes chat --query "reply just OK"'
```
4. Restore afterwards:
```bash
docker exec -u root {container id} cp /tmp/resolv.bak /etc/resolv.conf
```
## Expected Result
A single `hermes chat` turn should complete in a few seconds when the previous session worked, or fail fast (single clear "DNS timeout" / "cannot resolve host" message within a small budget) when DNS is stale. It should not silently retry three times, doing full DNS resolution on each attempt, before giving up.
## Actual Result
Under stale DNS the same chat query takes ~67 s to fail — Hermes prints three retry rows whose `Elapsed` timing grows because each retry re-resolves the upstream host:
- Attempt 1/3: `APIConnectionError` — Elapsed **31.23 s** (~2 DNS lookups × ~8 s each + connect)
- Attempt 2/3: `APIConnectionError` — Elapsed **38.26 s**
- Attempt 3/3: `APIConnectionError` — Elapsed **46.69 s**
- Total wall time for one query: **1 min 7 s**
Under a healthy resolver on the same container the same failing call takes **~29 s** (Elapsed 13 / 16 / 20 s) — so stale DNS more than doubles per-query latency. Multiplied across a typical multi-turn Hermes session (~15-20 tool-using turns), the end-to-end wait matches the user-reported ~20 minutes.
The measured DNS timeout inside the container under the stale configuration is **~8 s per hostname lookup** (`time getent hosts api.deepseek.com` and `time getent hosts inference.local` both return in ~8.07 s). This matches Hermes' per-attempt overhead pattern.
## Logs
```text
Stale-DNS reproduction (single hermes chat, 3 retries):
$ time getent hosts api.deepseek.com
real 0m8.063s
$ time getent hosts inference.local
real 0m8.075s
$ time timeout 120 hermes chat --query "just say OK"
Provider: custom Model: nvidia/nemotron-3-super-120b-a12b
Endpoint: https://inference.local/v1
Error: Connection error.
Elapsed: 31.23s Context: 2 msgs, ~4,246 tokens
Retrying in 2.8s (attempt 1/3)...
Provider: custom Model: nvidia/nemotron-3-super-120b-a12b
Endpoint: https://inference.local/v1
Error: Connection error.
Elapsed: 38.26s
Retrying in 4.3s (attempt 2/3)...
Provider: custom Model: nvidia/nemotron-3-super-120b-a12b
Endpoint: https://inference.local/v1
Error: Connection error.
Elapsed: 46.69s
API failed after 3 retries — Connection error.
Session: 20260804_105509_f5d759
Duration: 1m 4s
real 1m 7.121s
Baseline (embedded Docker DNS, same broken upstream) for comparison — same
3-retry pattern but each attempt Elapsed 13.62 / 16.04 / 20.47 s, total 29 s.
```
Contributor guide
Research direction
Start at the Hermes CLI path invoked by `hermes chat --query` and trace the shared upstream retry and DNS-resolution behavior described in the reproduction. Use the stale `/etc/resolv.conf` commands and timing measurements to reproduce the delay, then verify that stale-DNS failures fail within a small budget instead of repeating full DNS retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, typescript
- Domain
- cli, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100