NVIDIA / NVIDIA/NemoClaw

fix(mcp): scope Hermes MCP status reconciliation to the queried server

Open
#11,118 1 comment 0 reactions 0 assignees View on GitHub
area: cli integration: hermes
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 43m
Merged PRs (30d)
718

Description

### Investigation Summary

- `nemoclaw mcp status ` on a Hermes sandbox with two or more managed MCP servers can report a perfectly healthy, correctly-configured server as unregistered ("Hermes MCP config does not match persisted managed intent"), even though its entry in `/sandbox/.hermes/config.yaml` is valid and the Hermes gateway serves it correctly. Restarting the "broken" server appears to fix it, but breaks whichever sibling server was last queried — querying/restarting servers becomes a permanent game of musical chairs where exactly one managed server appears healthy at a time.
- Verified against a live two-server Hermes sandbox (`homeassistant` + `semaphore`): the low-level transaction helper (`hermes-mcp-config-transaction.py inspect`), invoked directly with a canonical, revision-agnostic payload for both servers, reports `{"ok": true, "state": "matched"}` for both — proving the on-disk config and the running gateway are correct. The false "mismatch" is produced entirely on the NemoClaw host side.
- Root cause identified: `statusMcpBridge()` only freshly observes a live OpenShell credential revision for the single named server being queried, but the resulting partial revision map is passed into `inspectHermesMcpRuntimeIntent()` without also scoping its `entries`/`managedServerNames` — so the reconciliation check silently defaults to every bridge entry on the sandbox, holding the queried server to an exact revision match while every unobserved sibling passes unconditionally.
- Fix implemented and verified: scoping both `entries` and `managedServerNames` to the queried server when one is named. Confirmed with a new regression test that fails on unpatched code and passes with the fix, plus the existing 463-test MCP bridge suite passing with no regressions.
- Known follow-up (separate, not yet resolved): even with this fix applied, a single server's status can still flip to "unregistered" if enough wall-clock time passes since its last write, because OpenShell's periodic credential-revision refresh worker can legitimately rotate a provider's live revision past what's committed on disk. This is a distinct design question (exact-match intent vs. tolerable rotation) tracked separately.

### Description

**What happened:** On a Hermes sandbox with two or more managed MCP servers, running `nemoclaw mcp status ` for a specific server can falsely report that server as unregistered, with the detail "Hermes MCP config does not match persisted managed intent" — even though its entry in `/sandbox/.hermes/config.yaml` is fully valid and the Hermes gateway serves it correctly.

Root cause: `statusMcpBridge()` observes a fresh, live OpenShell credential revision only for the one named server being queried (one sandbox exec per query, by design). That partial revision map is then passed into `inspectHermesMcpRuntimeIntent()` without also scoping its `entries`/`managedServerNames`, so the reconciliation call silently defaults to checking every bridge entry registered on the sandbox — not just the one server actually observed.

Consequence: the queried server's expected config is built with an exact, live-observed credential revision (requiring an exact match, no tolerance), while every other, unobserved managed server gets a canonical revision-agnostic expectation (tolerating any validly-shaped revision on disk). Whichever server was queried or restarted most recently is therefore always the fragile one, and every other server always reports as matched regardless of its real state — producing a permanent game of musical chairs where only one server appears healthy at a time.

**What I expected:** `mcp status ` should report a server's own registration state independently of which other managed servers exist or were recently queried. Checking one server should never make an unrelated sibling server look broken or fixed.

**Proposed fix:** I have a working patch and a passing regression test (proven to fail without the fix, pass with it, and validated against 463 existing tests with no regressions) — see linked PR.

### Reproduction Steps

1. On a Hermes sandbox, add two managed MCP servers:
nemoclaw mcp add server-a --url --env SERVER_A_TOKEN
nemoclaw mcp add server-b --url --env SERVER_B_TOKEN
2. export SERVER_A_TOKEN=... ; nemoclaw mcp restart server-a ; unset SERVER_A_TOKEN
3. nemoclaw mcp status server-b --json
→ adapter.registered: false, "Hermes MCP config does not match persisted managed intent"
(server-b was never touched by step 2, yet it is now reported broken)
4. export SERVER_B_TOKEN=... ; nemoclaw mcp restart server-b ; unset SERVER_B_TOKEN
5. nemoclaw mcp status server-a --json
→ now server-a flips to adapter.registered: false — the pattern repeats indefinitely,
with whichever server was queried least recently always reporting broken.

### Environment

- OS: Ubuntu 24.04 (remote server, accessed via SSH)
- Hardware: x86_64 workstation (10 vCPU / 7.7 GiB RAM, self-hosted)
- Node.js: v22.23.2
- Docker: 29.7.2
- NemoClaw: v0.0.120-4-gafb234266 (git checkout of NVIDIA/NemoClaw main)
- Agent: Hermes Agent v0.20.6
- OpenShell: 0.0.106

### Debug Output

### Reproduction 1 — cross-server status flip (mcp status )

```
$ nemoclaw mcp status server-a --json

{
"provider": {
"name": "hermes-mcp-server-a-",
"credentialResolution": { "ok": null, "detail": "probe curl exited 56" }
},
"adapter": { "registered": true }
}

$ nemoclaw mcp status server-b --json
{
"provider": {
"name": "hermes-mcp-server-b-",
"credentialResolution": {
"ok": null,
"detail": "probe skipped: the managed agent adapter does not match the current credential revision"
}
},
"adapter": {
"registered": false,
"detail": "Hermes MCP config does not match persisted managed intent"
}
}
```

### Reproduction 2 — low-level ground truth proves the disk/gateway are correct
```
$ openshell sandbox exec --name --no-tty -- \
/opt/hermes/.venv/bin/python /usr/local/lib/nemoclaw/hermes-mcp-config-transaction.py inspect \
--payload '{
"present": {
"server-a": {
"url": "https://mcp-gw.example.test/mcp-a",
"enabled": true, "timeout": 120, "connect_timeout": 60,
"tools": {"resources": true, "prompts": true},
"headers": {"Authorization": "Bearer openshell:resolve:env:SERVER_A_TOKEN"}
},
"server-b": {
"url": "https://mcp-gw.example.test/mcp-b",
"enabled": true, "timeout": 120, "connect_timeout": 60,
"tools": {"resources": true, "prompts": true},
"headers": {"Authorization": "Bearer openshell:resolve:env:SERVER_B_TOKEN"}
}
},
"absent": []
}'
{"ok": true, "state": "matched"}
```

### Reproduction 3 — actual on-disk config, both servers present and well-formed
```
$ openshell sandbox exec --name --no-tty -- cat /sandbox/.hermes/config.yaml
mcp_servers:
server-a:
url: https://mcp-gw.example.test/mcp-a
enabled: true
timeout: 120
connect_timeout: 60
tools:
resources: true
prompts: true
headers:
Authorization: Bearer openshell:resolve:env:v_SERVER_A_TOKEN
server-b:
url: https://mcp-gw.example.test/mcp-b
enabled: true
timeout: 120
connect_timeout: 60
tools:
resources: true
prompts: true
headers:
Authorization: Bearer openshell:resolve:env:v_SERVER_B_TOKEN
```

### Environment version confirmation

$ nemoclaw status
OpenShell: 0.0.106 (docker)
Agent: Hermes Agent v0.20.6

### Logs

```shell

```

### Checklist

- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate

Contributor guide

Open the contributing guide

Research direction

Start by reading statusMcpBridge() and inspectHermesMcpRuntimeIntent(), then examine the new regression test described in the issue. Reproduce the two-server status scenario if possible and verify that the queried server is reconciled independently; run the MCP bridge suite and confirm all 463 tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
backend, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.