anthropics / anthropics/sandbox-runtime

Bug: the mux front-end TCP port is shared across Node `cluster` workers, violating srt's own per-process proxy model → intermittent 407 in restricted mode

Đang mở
#458 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
TypeScript
Star
5.2k
Fork
439
Merge trung bình
2 ngày 9 phút
Pull request đã merge (30 ngày)
13

Mô tả

## Summary

srt is built around a **one-process-one-proxy** model: the mux backend binds a **pid-scoped** unix socket, and each process mints its **own** random `proxyAuthToken` (the code comment even says the token is set *"only when this process owns the proxy"*). But the mux **front-end** is bound with a plain `server.listen(port, '127.0.0.1')` — no `exclusive` flag. Under Node's `cluster` module (how PM2 `exec_mode: cluster` runs an app), `net.Server.listen()` is intercepted and the listen handle is **shared across all workers**, and the primary round-robins connections across them.

So a sandboxed command spawned by **worker A** (its env carries **A's** token) connects to the shared mux port, gets round-robined to **worker B**, and B's proxy checks A's token against **B's** `proxyAuthToken` → mismatch → 407:

```
HTTP/1.1 407 Proxy Authentication Required
Proxy-Authenticate: Basic realm="srt"
```

This is not a missing feature — it is an **internal inconsistency**: the front-end port sharing contradicts the per-process isolation the rest of the module (pid-scoped backend socket, per-process token) already assumes. The failure rate scales as ≈ `(N-1)/N` for `N` workers (e.g. ~67% at `N = 3` in the reproduction below); a retry often lands on the right worker and succeeds. Single-process (fork / 1-worker) deployments are unaffected.

## Root cause

**srt's design intent is per-process isolation.** Two places make this explicit:

1. The mux **backend** binds a **pid-scoped** unix socket — `sandbox/mux-proxy.js`:
```js
function unixSocketPath() {
return join(tmpdir(), `srt-mux-${process.pid}-${(backendSeq++).toString(36)}.sock`);
}
```
2. The **auth token is per-process**, and the comment frames it as process-owned — `sandbox/sandbox-manager.js`:
```js
// The auth token is only set when this process owns the proxy; an
// external proxy (config.network.httpProxyPort) handles its own auth ...
proxyAuthToken = config.network.httpProxyPort !== undefined
? undefined
: randomBytes(16).toString('hex'); // distinct per worker
```

**But the front-end binding breaks that assumption under `cluster`.** In `startMuxProxyServer()` — `sandbox/sandbox-manager.js`:
```js
await listenInRange(mux.server, p => mux.server.listen(p, '127.0.0.1'), portRange, ...);
// non-Windows: portRange is undefined -> listens on 127.0.0.1:0, no `exclusive` flag
```
`net.Server.listen()` under Node `cluster` defaults to `exclusive: false`, so the primary **shares one listen handle across all workers**. Every worker therefore reports the **same** `muxPort`, which each worker then bakes into the token URL for the commands *it* spawns — `sandbox/sandbox-utils.js`:
```js
const auth = proxyAuthToken ? `srt:${proxyAuthToken}@` : '';
// -> http_proxy=http://srt:@localhost:
```

Now worker A's command connects to the shared port, is round-robined to worker B, and B's auth check rejects A's token — `sandbox/http-proxy.js`:
```js
const checkAuth = (got) => {
if (!options.proxyAuthToken) return { ok: true };
const m = /^basic\s+([a-z0-9+/=]+)\s*$/i.exec(got ?? '');
if (!m) return { ok: false };
const decoded = Buffer.from(m[1], 'base64').toString('utf8');
const sep = decoded.indexOf(':');
if (sep <= 0 || decoded.slice(sep + 1) !== options.proxyAuthToken) { // A's token !== B's token
return { ok: false };
}
return { ok: true, encodedCommand: encodedCommandFromProxyUser(decoded.slice(0, sep)) };
};
```
→ 407, emitted at both the CONNECT and plain-HTTP paths with `realm="srt"`.

The backend is isolated per pid and the token is minted per process, but the **front-end port is the one piece that leaks across processes** — so the design's own invariant (a command always reaches the proxy that issued its token) is violated whenever srt runs under `cluster`.

## Minimal reproduction (pure `@anthropic-ai/sandbox-runtime`, no wrapper)

```js
// repro.mjs — run: node repro.mjs (in a project that has @anthropic-ai/sandbox-runtime installed)
import cluster from "node:cluster";
import { spawn } from "node:child_process";
import os from "node:os";
import { SandboxManager } from "@anthropic-ai/sandbox-runtime";
const N = 3, HOST = "example.com";
if (cluster.isPrimary) {
let d = 0; for (let i = 0; i < N; i++) cluster.fork();
cluster.on("exit", () => { if (++d === N) process.exit(0); });
} else {
await SandboxManager.initialize({
network: { allowedDomains: [HOST], deniedDomains: [] },
filesystem: { allowWrite: [os.tmpdir()], denyWrite: [], denyRead: [] },
}, undefined, false);
const wrun = (cmd) => new Promise(async (res) => {
const { argv, env } = await SandboxManager.wrapWithSandboxArgv(cmd, undefined, undefined, undefined);
const p = spawn(argv[0], argv.slice(1), { shell: false, env });
let o = ""; p.stdout.on("data", d => o += d); p.stderr.on("data", d => o += d); p.on("close", () => res(o));
});
const disc = await wrun("printenv http_proxy");
const port = (disc.match(/@[^:]+:(\d+)/) || [])[1];
const tok = (disc.match(/:([0-9a-f]{8})[0-9a-f]*@/) || [])[1];
const cls = (o) => /407|realm="srt"/i.test(o) ? "407" : /< HTTP\/1\.1 200/.test(o) ? "OK" : "OTHER";
const c = {};
for (let r = 0; r < 3; r++) {
const rs = await Promise.all(Array.from({ length: 8 }, () => wrun(`curl -sv --max-time 6 http://${HOST}/ 2>&1`)));
for (const o of rs) c[cls(o)] = (c[cls(o)] ?? 0) + 1;
}
console.error(`worker pid=${process.pid} muxPort=${port} token8=${tok} => ${JSON.stringify(c)}`);
process.exit(0);
}
```

Observed on **0.0.67** (3 workers):
```
worker pid=47253 muxPort=59010 token8=cef7c280 => {"407":16,"OK":8}
worker pid=47251 muxPort=59010 token8=11a649f1 => {"407":13,"OK":11}
worker pid=47252 muxPort=59010 token8=c8ed79fd => {"407":15,"OK":9}
```
All workers share **muxPort 59010**; each has a **distinct token**; ~61% (44/72) get 407. Running the same processes standalone (not via `cluster`) gives distinct ports and **0 × 407**.

## Still present on 0.0.71 (current `latest`)

The same `repro.mjs`, run against a clean install of **0.0.71**, reproduces identically:
```
worker pid=8502 muxPort=53775 token8=f22dc2b0 => {"407":17,"OK":5,"OTHER":2}
worker pid=8501 muxPort=53775 token8=f8ef7089 => {"407":14,"OK":7,"OTHER":3}
worker pid=8503 muxPort=53775 token8=409a04e9 => {"407":17,"OK":5,"OTHER":2}
```
All three workers again share one `muxPort` (53775) with distinct tokens; ~67% (48/72) get 407.

The relevant code is byte-for-byte unchanged from 0.0.67 to 0.0.71: the `mux.server.listen(p, '127.0.0.1')` bind (no `exclusive`), the pid-scoped backend socket, the per-worker `randomBytes(16)` token, and the `checkAuth` token comparison. The only proxy-path change across 0.0.68–0.0.71 is that `checkAuth` now returns `{ ok, encodedCommand }` (encoding the command into the proxy *user* field for attribution) — orthogonal to this bug; the password field is still strictly compared against the per-worker token, so cross-worker delivery still 407s. (Note 0.0.71 also added the same username/password auth to the SOCKS front-end, so a SOCKS deployment under `cluster` will exhibit the same cross-worker 407.)

## Confirming the shared handle is the trigger

A minimal `net` server under `cluster` shows the handle-sharing directly, and that `exclusive: true` fixes it:

```
listen({ port: 0, host: "127.0.0.1", exclusive: false }) -> worker ports [58956,58956,58956,58956] (distinct=1, shared)
listen({ port: 0, host: "127.0.0.1", exclusive: true }) -> worker ports [58957,58958,58959,58960] (distinct=4, private)
```

## Suggested fix

Both options **restore the per-process isolation the rest of the module already assumes**, and neither changes the exposure surface — the front-end stays bound to `127.0.0.1` either way; `exclusive` only controls whether sibling `cluster` workers share the socket, not who can reach it (the token remains the access control).

**Option 1 (minimal, verified):** bind the mux front-end with `exclusive: true` so each worker gets its own handle/port even under `cluster`:
```js
// sandbox-manager.js startMuxProxyServer()
await listenInRange(
mux.server,
p => mux.server.listen({ port: p, host: '127.0.0.1', exclusive: true }),
portRange, ...
);
```
The `exclusive: true` check above confirms this yields per-worker distinct ports under `cluster`. Each worker then injects its own `muxPort` into its own commands → no cross-worker delivery → no 407. Keeps the current TCP architecture.

**Option 2 (matches the existing design most closely):** bind the mux front-end on a **pid-scoped unix socket**, exactly as the mux *backend* already does (`srt-mux-${process.pid}-…​.sock`). Node `cluster` does not share pipe servers bound to distinct paths, so workers stay isolated by construction — and the front-end would then follow the same per-pid convention as the backend.

## Impact & workaround

Any consumer running srt in allowlist/restricted mode under Node `cluster` (PM2 `exec_mode: cluster` is a very common Node deployment) gets a high, intermittent 407 rate on **all** sandboxed egress — effectively unusable in cluster + restricted mode. Current workaround is to run one process per listener (PM2 `fork` mode behind a front LB, or one process per container/replica), which is a non-trivial topology change for cluster-based fleets. Upgrading through the latest release (0.0.71) does **not** address it.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.