NVIDIA / NVIDIA/NemoClaw

[Bug] Regression of #1471: egress proxy "failed to resolve peer binary" / binary=- on OpenShell 0.0.44 (docker, Ubuntu 24.04)

Open
#5,962 4 comments 0 reactions 0 assignees View on GitHub
area: networking area: policy area: sandbox platform: ubuntu
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

### Agent Diagnostic

Investigated the egress failure end-to-end before filing.

- Searched existing issues and found #1471 ("failed to resolve peer binary" — closed as fixed-on-latest after the OpenShell pin moved to 0.0.36) and #1798 (related peer-attribution discussion).
- Reviewed the closing verification on #1471: peer-binary resolution was confirmed working on OpenShell 0.0.36 (macOS arm64) and 0.0.39 (Ubuntu 22.04 amd64). The maintainers asked to reopen/refile if the symptom resurfaces on any platform.
- Confirmed the symptom maps to the connection→PID lookup step (the proxy resolves the calling binary by scanning the entrypoint PID's /proc//net/tcp), which fails *before* policy/binary matching is reached — hence policy:- and deny-by-default, not an endpoint denial.
- Verified it is not a policy-content problem, not conntrack exhaustion, and not a credential/endpoint problem (host-direct curl to the same endpoint succeeds). Details below.

This is filed as a new issue (rather than only commenting on the now-closed #1471) per the maintainers' "reopen if it resurfaces" note and to enter normal triage.

### Description

## Summary

On OpenShell 0.0.44, the sandbox egress proxy again denies all new outbound CONNECTs with `failed to resolve peer binary` / `binary=-`, exactly as in #1471 — **and it persists across a full sandbox recreate** (fresh container, fresh entrypoint PID). A sandbox that egressed to a single allow-listed endpoint reliably for ~11 days suddenly began denying every connection; nothing short of moving the workload off the sandbox restored egress.

This was reported as a comment on the now-closed #1471, but is being filed as a fresh issue per the "reopen if the symptom resurfaces on any platform" note so it enters normal triage.

## Expected

The proxy resolves the calling binary, matches it against the policy allow-rule (`kloud_browser`), and permits the CONNECT — as verified working on OpenShell 0.0.36 / 0.0.39 in #1471.

## Actual

The proxy never resolves the calling binary (`binary=-`, `policy:-`), so the valid allow-rule never matches and it denies by default. Resolution fails at the connection→PID lookup step, before binary/policy matching is reached — identical to the original #1471 signature. The LLM/inference route is unaffected (different egress path).

## Why this looks like a regression (not the original, already-fixed bug)

Per #1471 the fix landed in OpenShell ≥ 0.0.36. This host is on 0.0.44 and still hits `binary=-`, **including on a brand-new container**: a forced rebuild recreated the sandbox as a fresh container (new container ID, new entrypoint PID), and the very first egress attempt from that fresh container produced the same denial. A stale/recycled entrypoint PID alone cannot explain a fresh container failing immediately, so the resolution path appears not to be working in this docker-on-Linux setup.

Note: per @prekshivyas's analysis on #1471, OpenShell PR #956 (pid-1 netns fallback) is scoped to supervisor helpers and is structurally not the fix for workload-originated CONNECTs; resolution on 0.0.36 worked because the workload socket was visible in the entrypoint PID's `/proc//net/tcp`. This report suggests that visibility is not holding on 0.0.44 in this environment.

### Reproduction Steps

1. Run a sandbox with a policy that allow-lists a single outbound HTTPS endpoint (allow-rule `kloud_browser`).
2. Leave it running under normal load (degradation observed over ~11 days of uptime).
3. From inside the sandbox, issue an outbound HTTPS CONNECT to the allow-listed endpoint via the proxy.
4. Observe: every new CONNECT is DENIED with `failed to resolve peer binary` / `binary=-` / `policy:-`.
5. Attempt recovery (see "What was tried" in Logs) — including a forced rebuild onto a fresh container — and observe the very first CONNECT from the fresh container fails identically.

Reproduces immediately on the freshly recreated container; does not require waiting for the 11-day degradation again.

### Environment

- NemoClaw: v0.0.55
- OpenShell: 0.0.44 (≥ 0.0.36, i.e. should contain the #1471 fix)
- Sandbox agent: OpenClaw v2026.5.22
- Driver: docker (Docker 29.5.2)
- Host OS: Ubuntu 24.04.4 LTS, kernel 6.17.0-1018-azure (cloud VM)
- Inference: gateway route (unaffected throughout)
- Sandbox uptime when it broke: ~11 days

### Debug Output

```shell
Can provide full `nemoclaw doctor --json` and `nemoclaw debug --quick` output on request (will sanitize host/endpoint/credential details before posting). Key already-gathered diagnostics are summarized in the Logs section below.

- `nf_conntrack_count`: 161 of 262144 (not exhausted)
- Fresh container after forced rebuild: new container ID, new entrypoint PID, OpenClaw v2026.5.22 on OpenShell 0.0.44 — first CONNECT still `binary=-`.
```

### Logs

```shell
Denied CONNECT (binary never resolves):

[sandbox] [OCSF] [ocsf] NET:OPEN [MED] DENIED -(0) -> :443
[policy:- engine:opa]
[reason:failed to resolve peer binary: No ESTABLISHED TCP connection found for port ]

For contrast, the same binary / destination / policy resolved correctly *before* the break:

[sandbox] [OCSF] [ocsf] NET:OPEN [INFO] ALLOWED /usr/bin/curl(349086) -> :443
[policy:kloud_browser engine:opa]

### What was tried (none restored egress)

1. `nemoclaw recover` (x2)
2. `docker restart `
3. Full gateway kill + relaunch (preserving env)
4. Full VM reboot
5. `nemoclaw rebuild --force` (fresh container)

After every one, the first outbound CONNECT denied with the same `binary=-` reason.

### Ruled out

- **Policy content** — allow-rule present and correct (`policy:kloud_browser`); denial is `policy:-` because the binary never resolves, not because the destination is disallowed.
- **conntrack exhaustion** — `nf_conntrack_count` 161 of 262144.
- **Credentials / endpoint** — host-direct curl to the same endpoint returns 200; failure is purely the in-sandbox proxy's peer-binary resolution.
- **Transient state** — survives container restart, gateway restart, and a full host reboot.

### Workaround in use

Running the egress workload on the host, outside the sandbox proxy, restores connectivity (host → endpoint = 200), at the cost of in-sandbox policy enforcement and dashboard visibility.

### Distinguishing from policy_denied

Per the guidance on #1798, this is the peer-attribution failure class (`reason: failed to resolve peer binary` / `policy:-`), **not** a `policy_denied` body. Happy to attach a raw `curl -i -x ... --proxytunnel` capture of the deny body to confirm.
```

### Checklist

- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate

Contributor guide

Open the contributing guide

Research direction

Start by tracing the egress proxy's connection-to-PID lookup and compare the entrypoint PID's /proc//net/tcp visibility between OpenShell 0.0.39 and 0.0.44 on Docker Ubuntu. Reproduce with a fresh container and the allow-listed kloud_browser endpoint, then verify that the proxy resolves the calling binary and permits CONNECT instead of logging binary=- and policy:-.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux, typescript, ubuntu
Domain
infrastructure, networking, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.