[Bug] Regression of #1471: egress proxy "failed to resolve peer binary" / binary=- on OpenShell 0.0.44 (docker, Ubuntu 24.04)
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
### Agent Diagnostic
Investigated the egress failure end-to-end before filing.
- Searched existing issues and found #1471 ("failed to resolve peer binary" — closed as fixed-on-latest after the OpenShell pin moved to 0.0.36) and #1798 (related peer-attribution discussion).
- Reviewed the closing verification on #1471: peer-binary resolution was confirmed working on OpenShell 0.0.36 (macOS arm64) and 0.0.39 (Ubuntu 22.04 amd64). The maintainers asked to reopen/refile if the symptom resurfaces on any platform.
- Confirmed the symptom maps to the connection→PID lookup step (the proxy resolves the calling binary by scanning the entrypoint PID's /proc//net/tcp), which fails *before* policy/binary matching is reached — hence policy:- and deny-by-default, not an endpoint denial.
- Verified it is not a policy-content problem, not conntrack exhaustion, and not a credential/endpoint problem (host-direct curl to the same endpoint succeeds). Details below.
This is filed as a new issue (rather than only commenting on the now-closed #1471) per the maintainers' "reopen if it resurfaces" note and to enter normal triage.
### Description
## Summary
On OpenShell 0.0.44, the sandbox egress proxy again denies all new outbound CONNECTs with `failed to resolve peer binary` / `binary=-`, exactly as in #1471 — **and it persists across a full sandbox recreate** (fresh container, fresh entrypoint PID). A sandbox that egressed to a single allow-listed endpoint reliably for ~11 days suddenly began denying every connection; nothing short of moving the workload off the sandbox restored egress.
This was reported as a comment on the now-closed #1471, but is being filed as a fresh issue per the "reopen if the symptom resurfaces on any platform" note so it enters normal triage.
## Expected
The proxy resolves the calling binary, matches it against the policy allow-rule (`kloud_browser`), and permits the CONNECT — as verified working on OpenShell 0.0.36 / 0.0.39 in #1471.
## Actual
The proxy never resolves the calling binary (`binary=-`, `policy:-`), so the valid allow-rule never matches and it denies by default. Resolution fails at the connection→PID lookup step, before binary/policy matching is reached — identical to the original #1471 signature. The LLM/inference route is unaffected (different egress path).
## Why this looks like a regression (not the original, already-fixed bug)
Per #1471 the fix landed in OpenShell ≥ 0.0.36. This host is on 0.0.44 and still hits `binary=-`, **including on a brand-new container**: a forced rebuild recreated the sandbox as a fresh container (new container ID, new entrypoint PID), and the very first egress attempt from that fresh container produced the same denial. A stale/recycled entrypoint PID alone cannot explain a fresh container failing immediately, so the resolution path appears not to be working in this docker-on-Linux setup.
Note: per @prekshivyas's analysis on #1471, OpenShell PR #956 (pid-1 netns fallback) is scoped to supervisor helpers and is structurally not the fix for workload-originated CONNECTs; resolution on 0.0.36 worked because the workload socket was visible in the entrypoint PID's `/proc//net/tcp`. This report suggests that visibility is not holding on 0.0.44 in this environment.
### Reproduction Steps
1. Run a sandbox with a policy that allow-lists a single outbound HTTPS endpoint (allow-rule `kloud_browser`).
2. Leave it running under normal load (degradation observed over ~11 days of uptime).
3. From inside the sandbox, issue an outbound HTTPS CONNECT to the allow-listed endpoint via the proxy.
4. Observe: every new CONNECT is DENIED with `failed to resolve peer binary` / `binary=-` / `policy:-`.
5. Attempt recovery (see "What was tried" in Logs) — including a forced rebuild onto a fresh container — and observe the very first CONNECT from the fresh container fails identically.
Reproduces immediately on the freshly recreated container; does not require waiting for the 11-day degradation again.
### Environment
- NemoClaw: v0.0.55
- OpenShell: 0.0.44 (≥ 0.0.36, i.e. should contain the #1471 fix)
- Sandbox agent: OpenClaw v2026.5.22
- Driver: docker (Docker 29.5.2)
- Host OS: Ubuntu 24.04.4 LTS, kernel 6.17.0-1018-azure (cloud VM)
- Inference: gateway route (unaffected throughout)
- Sandbox uptime when it broke: ~11 days
### Debug Output
```shell
Can provide full `nemoclaw doctor --json` and `nemoclaw debug --quick` output on request (will sanitize host/endpoint/credential details before posting). Key already-gathered diagnostics are summarized in the Logs section below.
- `nf_conntrack_count`: 161 of 262144 (not exhausted)
- Fresh container after forced rebuild: new container ID, new entrypoint PID, OpenClaw v2026.5.22 on OpenShell 0.0.44 — first CONNECT still `binary=-`.
```
### Logs
```shell
Denied CONNECT (binary never resolves):
[sandbox] [OCSF] [ocsf] NET:OPEN [MED] DENIED -(0) -> :443
[policy:- engine:opa]
[reason:failed to resolve peer binary: No ESTABLISHED TCP connection found for port ]
For contrast, the same binary / destination / policy resolved correctly *before* the break:
[sandbox] [OCSF] [ocsf] NET:OPEN [INFO] ALLOWED /usr/bin/curl(349086) -> :443
[policy:kloud_browser engine:opa]
### What was tried (none restored egress)
1. `nemoclaw recover` (x2)
2. `docker restart `
3. Full gateway kill + relaunch (preserving env)
4. Full VM reboot
5. `nemoclaw rebuild --force` (fresh container)
After every one, the first outbound CONNECT denied with the same `binary=-` reason.
### Ruled out
- **Policy content** — allow-rule present and correct (`policy:kloud_browser`); denial is `policy:-` because the binary never resolves, not because the destination is disallowed.
- **conntrack exhaustion** — `nf_conntrack_count` 161 of 262144.
- **Credentials / endpoint** — host-direct curl to the same endpoint returns 200; failure is purely the in-sandbox proxy's peer-binary resolution.
- **Transient state** — survives container restart, gateway restart, and a full host reboot.
### Workaround in use
Running the egress workload on the host, outside the sandbox proxy, restores connectivity (host → endpoint = 200), at the cost of in-sandbox policy enforcement and dashboard visibility.
### Distinguishing from policy_denied
Per the guidance on #1798, this is the peer-attribution failure class (`reason: failed to resolve peer binary` / `policy:-`), **not** a `policy_denied` body. Happy to attach a raw `curl -i -x ... --proxytunnel` capture of the deny body to confirm.
```
### Checklist
- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate
Contributor guide
Research direction
Start by tracing the egress proxy's connection-to-PID lookup and compare the entrypoint PID's /proc//net/tcp visibility between OpenShell 0.0.39 and 0.0.44 on Docker Ubuntu. Reproduce with a fresh container and the allow-listed kloud_browser endpoint, then verify that the proxy resolves the calling binary and permits CONNECT instead of logging binary=- and policy:-.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, linux, typescript, ubuntu
- Domain
- infrastructure, networking, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100