anthropics / anthropics/sandbox-runtime

whichSync Node fallback folds timeout/ENOENT/EACCES into "Shell 'X' not found in PATH" — transient CPU starvation kills execs with a misleading error (absolute paths are not exempt)

Aperta
#548 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
TypeScript
Stelle
5.2k
Fork
441
Merge medio
1g 19h
PR unite (30g)
12

Descrizione

## Environment

- `@anthropic-ai/sandbox-runtime@0.0.73` (pinned; I diffed `dist/utils/which.js` and the wrap throw site in `0.0.76` — byte-identical, so this applies to the latest release too)
- Node v26 (also seen on Node v22 in production), Linux container, **1 CPU**
- Embedding SRT via `SandboxManager.wrapWithSandbox` / `wrapCommandWithSandboxLinux` (public API, README usage)

## What happens

`dist/utils/which.js` resolves the shell by shelling out to `which` with a 1-second wall-clock timeout, then keeps only `status === 0 && stdout`:

```js
const result = spawnSync('which', [bin], {
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'ignore'],
timeout: 1000,
});
if (result.status === 0 && result.stdout) return result.stdout.trim();
return null;
```

`result.error` (ETIMEDOUT / ENOENT / EACCES), `result.signal`, and the non-zero `status` are all discarded. `linux-sandbox-utils.js` then treats `null` as one thing:

```js
const shellName = binShell || 'bash';
const shell = whichSync(shellName);
if (!shell) {
throw new Error(`Shell '${shellName}' not found in PATH`);
}
```

So four completely different failures produce one message:

| Actual condition | raw `spawnSync` result | SRT reports |
|---|---|---|
| `which` runs slow (CPU starvation; the 1s deadline expires) | `status: null`, `signal: SIGTERM`, `error: ETIMEDOUT` (~1003 ms) | `Shell 'bash' not found in PATH` |
| `which` binary missing | `status: null`, `error: ENOENT` (0 ms) | same |
| `which` not executable | `status: null`, `error: EACCES` | same |
| shell genuinely absent | `status: 1` | same (correct) |

On a 1-CPU container under CPU saturation, a `which` invocation that normally takes a few milliseconds gets starved past the 1s wall-clock deadline and **every sandboxed exec fails pre-spawn with an error that misstates the system**: bash exists, PATH is fine — the lookup tool itself timed out. We hit 73 such failures across 17 requests in a 2-hour window during a rollout (production incident); the log message pointed everyone at "missing shell / broken PATH" while the real trigger was transient CPU starvation.

Two aggravating details:

1. **Absolute paths don't bypass the lookup.** `binShell: '/bin/bash'` still goes through `whichSync`, so under a starving `which` the error literally claims `/bin/bash` is "not found in PATH" — a self-contradicting message.
2. **The timeout is wall-clock, not CPU.** `timeout: 1000` measures elapsed time including scheduler queueing, so under load the deadline is consumed before the child even runs.

## Why the Bun path never sees this

`whichSync` prefers `Bun.which`, which is an in-process lookup — no fork, no timeout, no failure mode. The Bun test suite (`test/utils/which.test.ts`) asserts `typeof globalThis.Bun === 'object'`, so the Node fallback only has the hand-run `which-node-test.mjs`, which covers the happy paths only (finds `ls`/`bash`, null for a bogus name). None of the four failure classes above is exercised in CI — which is presumably why this has survived.

## Minimal reproduction

Self-contained (uses only the installed package; Linux or macOS host):

```js
// repro.mjs — npm i @anthropic-ai/sandbox-runtime@0.0.73 && node repro.mjs
import { spawnSync } from "node:child_process";
import { mkdtempSync, mkdirSync, writeFileSync, chmodSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";

const { whichSync } = await import(
"@anthropic-ai/sandbox-runtime/dist/utils/which.js"
);
const { wrapCommandWithSandboxLinux } = await import(
"@anthropic-ai/sandbox-runtime/dist/sandbox/linux-sandbox-utils.js"
);

// Fixture: a `which` that would succeed, but takes 2s (> the 1s deadline).
// NOTE: /bin/sleep must be absolute — the child's PATH only contains this dir.
const fix = mkdtempSync(join(tmpdir(), "srt-which-"));
const bin = join(fix, "bin");
mkdirSync(bin);
const fakeWhich = join(bin, "which");
writeFileSync(fakeWhich, "#!/bin/sh\n/bin/sleep 2\necho /bin/bash\nexit 0\n");
chmodSync(fakeWhich, 0o755);

function probe(label, pathEnv, target) {
const prev = process.env.PATH;
process.env.PATH = pathEnv;
const t0 = Date.now();
const r = whichSync(target);
process.env.PATH = prev;
console.log(`whichSync('${target}') [${label}] ${Date.now() - t0}ms ->`, r);
}

async function wrap(label, pathEnv, binShell) {
const prev = process.env.PATH;
process.env.PATH = pathEnv;
try {
const out = await wrapCommandWithSandboxLinux({
command: "echo hello",
readConfig: { denyOnly: ["/proc/self/environ"] },
...(binShell !== undefined ? { binShell } : {}),
});
console.log(`[${label}] OK: ${out.slice(0, 60)}...`);
} catch (e) {
console.log(`[${label}] THREW: ${e.message}`);
} finally {
process.env.PATH = prev;
}
}

probe("normal PATH", process.env.PATH, "bash"); // "/bin/bash"
probe("starving which", bin, "bash"); // null after ~1000ms (SIGTERM folded away)
probe("starving which, absolute", bin, "/bin/bash"); // null — absolute path not exempt
await wrap("normal PATH (control)", process.env.PATH, undefined); // OK
await wrap("starving which", bin, undefined); // Shell 'bash' not found in PATH
await wrap("starving which, binShell=/bin/bash", bin, "/bin/bash"); // Shell '/bin/bash' not found in PATH
await wrap("no which on PATH", fix, undefined); // same message (ENOENT folded away)
```

Output on my machine (Node v26, pinned 0.0.73):

```
whichSync('bash') [normal PATH] 2ms -> /bin/bash
whichSync('bash') [starving which] 1001ms -> null
whichSync('/bin/bash') [starving which, absolute] 1002ms -> null
[normal PATH (control)] OK: bwrap --new-session --die-with-parent --bind / ...
[starving which] THREW: Shell 'bash' not found in PATH
[starving which, binShell=/bin/bash] THREW: Shell '/bin/bash' not found in PATH
[no which on PATH] THREW: Shell 'bash' not found in PATH
```

## Suggested fixes (any of these would help; the first is the real fix)

1. **Resolve in-process in the Node fallback** instead of forking `which` — walk `process.env.PATH` with `fs.statSync`/access checks, matching `Bun.which` semantics. This eliminates the fork, the 1s wall-clock deadline, and the entire class of "the lookup tool itself failed" errors. A per-process or TTL cache would also remove the per-exec fork cost on the happy path.
2. **Preserve the failure class**: return/throw `error.code`, `signal`, and `status` so embedders can distinguish "timed out" from "not installed" instead of getting a message that misstates the system.
3. **Skip the lookup for absolute `binShell`**: a `binShell` containing a path separator should be used directly (with an executability check), not routed through `which` — the current behavior reports `/bin/bash` as "not found in PATH", which contradicts itself.

Happy to turn the repro into a PR for (1) + (3) if the approach sounds right.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.