NVIDIA / NVIDIA/NemoClaw

[Ubuntu 26.04][Agent&Skills] hermes chat stalls after host network change (stale sandbox DNS)

Open
#8,219 0 comments 0 reactions 0 assignees View on GitHub
area: networking area: performance area: sandbox integration: hermes needs: unblock NV QA platform: ubuntu
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

After the host machine's network changes (e.g. laptop switched from home Wi-Fi to office network) without restarting or reconnecting the sandbox container, the Hermes CLI's built-in "3 retries with backoff" wraps each upstream call in a fresh DNS lookup. When the container's `/etc/resolv.conf` still points at the pre-switch resolvers (Docker snapshots `resolv.conf` at container creation and does not refresh it when the host's DNS changes), every retry stalls on DNS timeouts. A single `hermes chat --query` grows from a few seconds to ~67 s, and a typical multi-turn interactive session (~15-20 tool-using turns) compounds to ~20 min end-to-end. The original user report showed the same behaviour independent of the configured provider — both a cloud provider (`api.deepseek.com`) and a locally-deployed Ollama model exhibited the slowdown, confirming the delay is in the shared retry+DNS layer rather than any single provider.

**Platform scope:** Reproduced on Ubuntu 26.04 (x86_64) only; other platforms not tested.
**Regression:** Unknown — earlier versions not tested.

## Environment

```text
Device: Ubuntu 26.04 LTS server (x86_64)
OS: Ubuntu 26.04 LTS (kernel 7.0.0-27-generic)
Architecture: x86_64
Node.js: v22.23.1
npm: 10.9.8
Docker: 29.6.1
OpenShell CLI: 0.0.72
NemoClaw: v0.0.101
OpenClaw: Hermes agent v0.17.0 (2026.6.19)
```

## Steps to Reproduce

1. On a healthy host with working DNS, onboard a Hermes sandbox (following https://docs.nvidia.com/nemoclaw/latest/get-started/quickstart-hermes.html) and confirm `hermes chat` responds within a few seconds.

2. Simulate a network change without restarting the sandbox container — the simplest deterministic proxy for the real "home to office" transition is to point the container's `/etc/resolv.conf` at an unreachable nameserver:

```bash
docker exec -u root {container id} sh -c \
'cp /etc/resolv.conf /tmp/resolv.bak && \
printf "nameserver 203.0.113.10\noptions timeout:2 attempts:2 ndots:0\n" > /etc/resolv.conf'
```

3. Measure a single `hermes chat` inside the sandbox:

```bash
time docker exec -u sandbox {container id} sh -c \
'cd /sandbox && HOME=/sandbox hermes chat --query "reply just OK"'
```

4. Restore afterwards:

```bash
docker exec -u root {container id} cp /tmp/resolv.bak /etc/resolv.conf
```

## Expected Result

A single `hermes chat` turn should complete in a few seconds when the previous session worked, or fail fast (single clear "DNS timeout" / "cannot resolve host" message within a small budget) when DNS is stale. It should not silently retry three times, doing full DNS resolution on each attempt, before giving up.

## Actual Result

Under stale DNS the same chat query takes ~67 s to fail — Hermes prints three retry rows whose `Elapsed` timing grows because each retry re-resolves the upstream host:

- Attempt 1/3: `APIConnectionError` — Elapsed **31.23 s** (~2 DNS lookups × ~8 s each + connect)
- Attempt 2/3: `APIConnectionError` — Elapsed **38.26 s**
- Attempt 3/3: `APIConnectionError` — Elapsed **46.69 s**
- Total wall time for one query: **1 min 7 s**

Under a healthy resolver on the same container the same failing call takes **~29 s** (Elapsed 13 / 16 / 20 s) — so stale DNS more than doubles per-query latency. Multiplied across a typical multi-turn Hermes session (~15-20 tool-using turns), the end-to-end wait matches the user-reported ~20 minutes.

The measured DNS timeout inside the container under the stale configuration is **~8 s per hostname lookup** (`time getent hosts api.deepseek.com` and `time getent hosts inference.local` both return in ~8.07 s). This matches Hermes' per-attempt overhead pattern.

## Logs

```text
Stale-DNS reproduction (single hermes chat, 3 retries):

$ time getent hosts api.deepseek.com
real 0m8.063s

$ time getent hosts inference.local
real 0m8.075s

$ time timeout 120 hermes chat --query "just say OK"
Provider: custom Model: nvidia/nemotron-3-super-120b-a12b
Endpoint: https://inference.local/v1
Error: Connection error.
Elapsed: 31.23s Context: 2 msgs, ~4,246 tokens
Retrying in 2.8s (attempt 1/3)...

Provider: custom Model: nvidia/nemotron-3-super-120b-a12b
Endpoint: https://inference.local/v1
Error: Connection error.
Elapsed: 38.26s
Retrying in 4.3s (attempt 2/3)...

Provider: custom Model: nvidia/nemotron-3-super-120b-a12b
Endpoint: https://inference.local/v1
Error: Connection error.
Elapsed: 46.69s
API failed after 3 retries — Connection error.

Session: 20260804_105509_f5d759
Duration: 1m 4s
real 1m 7.121s

Baseline (embedded Docker DNS, same broken upstream) for comparison — same
3-retry pattern but each attempt Elapsed 13.62 / 16.04 / 20.47 s, total 29 s.
```

Contributor guide

Open the contributing guide

Research direction

Start at the Hermes CLI path invoked by `hermes chat --query` and trace the shared upstream retry and DNS-resolution behavior described in the reproduction. Use the stale `/etc/resolv.conf` commands and timing measurements to reproduce the delay, then verify that stale-DNS failures fail within a small budget instead of repeating full DNS retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
cli, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.