anthropics / anthropics/sandbox-runtime
whichSync Node fallback folds timeout/ENOENT/EACCES into "Shell 'X' not found in PATH" — transient CPU starvation kills execs with a misleading error (absolute paths are not exempt)
- Lingua principale
- TypeScript
- Stelle
- 5.2k
- Fork
- 441
- Merge medio
- 1g 19h
- PR unite (30g)
- 12
Descrizione
## Environment
- `@anthropic-ai/sandbox-runtime@0.0.73` (pinned; I diffed `dist/utils/which.js` and the wrap throw site in `0.0.76` — byte-identical, so this applies to the latest release too)
- Node v26 (also seen on Node v22 in production), Linux container, **1 CPU**
- Embedding SRT via `SandboxManager.wrapWithSandbox` / `wrapCommandWithSandboxLinux` (public API, README usage)
## What happens
`dist/utils/which.js` resolves the shell by shelling out to `which` with a 1-second wall-clock timeout, then keeps only `status === 0 && stdout`:
```js
const result = spawnSync('which', [bin], {
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'ignore'],
timeout: 1000,
});
if (result.status === 0 && result.stdout) return result.stdout.trim();
return null;
```
`result.error` (ETIMEDOUT / ENOENT / EACCES), `result.signal`, and the non-zero `status` are all discarded. `linux-sandbox-utils.js` then treats `null` as one thing:
```js
const shellName = binShell || 'bash';
const shell = whichSync(shellName);
if (!shell) {
throw new Error(`Shell '${shellName}' not found in PATH`);
}
```
So four completely different failures produce one message:
| Actual condition | raw `spawnSync` result | SRT reports |
|---|---|---|
| `which` runs slow (CPU starvation; the 1s deadline expires) | `status: null`, `signal: SIGTERM`, `error: ETIMEDOUT` (~1003 ms) | `Shell 'bash' not found in PATH` |
| `which` binary missing | `status: null`, `error: ENOENT` (0 ms) | same |
| `which` not executable | `status: null`, `error: EACCES` | same |
| shell genuinely absent | `status: 1` | same (correct) |
On a 1-CPU container under CPU saturation, a `which` invocation that normally takes a few milliseconds gets starved past the 1s wall-clock deadline and **every sandboxed exec fails pre-spawn with an error that misstates the system**: bash exists, PATH is fine — the lookup tool itself timed out. We hit 73 such failures across 17 requests in a 2-hour window during a rollout (production incident); the log message pointed everyone at "missing shell / broken PATH" while the real trigger was transient CPU starvation.
Two aggravating details:
1. **Absolute paths don't bypass the lookup.** `binShell: '/bin/bash'` still goes through `whichSync`, so under a starving `which` the error literally claims `/bin/bash` is "not found in PATH" — a self-contradicting message.
2. **The timeout is wall-clock, not CPU.** `timeout: 1000` measures elapsed time including scheduler queueing, so under load the deadline is consumed before the child even runs.
## Why the Bun path never sees this
`whichSync` prefers `Bun.which`, which is an in-process lookup — no fork, no timeout, no failure mode. The Bun test suite (`test/utils/which.test.ts`) asserts `typeof globalThis.Bun === 'object'`, so the Node fallback only has the hand-run `which-node-test.mjs`, which covers the happy paths only (finds `ls`/`bash`, null for a bogus name). None of the four failure classes above is exercised in CI — which is presumably why this has survived.
## Minimal reproduction
Self-contained (uses only the installed package; Linux or macOS host):
```js
// repro.mjs — npm i @anthropic-ai/sandbox-runtime@0.0.73 && node repro.mjs
import { spawnSync } from "node:child_process";
import { mkdtempSync, mkdirSync, writeFileSync, chmodSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
const { whichSync } = await import(
"@anthropic-ai/sandbox-runtime/dist/utils/which.js"
);
const { wrapCommandWithSandboxLinux } = await import(
"@anthropic-ai/sandbox-runtime/dist/sandbox/linux-sandbox-utils.js"
);
// Fixture: a `which` that would succeed, but takes 2s (> the 1s deadline).
// NOTE: /bin/sleep must be absolute — the child's PATH only contains this dir.
const fix = mkdtempSync(join(tmpdir(), "srt-which-"));
const bin = join(fix, "bin");
mkdirSync(bin);
const fakeWhich = join(bin, "which");
writeFileSync(fakeWhich, "#!/bin/sh\n/bin/sleep 2\necho /bin/bash\nexit 0\n");
chmodSync(fakeWhich, 0o755);
function probe(label, pathEnv, target) {
const prev = process.env.PATH;
process.env.PATH = pathEnv;
const t0 = Date.now();
const r = whichSync(target);
process.env.PATH = prev;
console.log(`whichSync('${target}') [${label}] ${Date.now() - t0}ms ->`, r);
}
async function wrap(label, pathEnv, binShell) {
const prev = process.env.PATH;
process.env.PATH = pathEnv;
try {
const out = await wrapCommandWithSandboxLinux({
command: "echo hello",
readConfig: { denyOnly: ["/proc/self/environ"] },
...(binShell !== undefined ? { binShell } : {}),
});
console.log(`[${label}] OK: ${out.slice(0, 60)}...`);
} catch (e) {
console.log(`[${label}] THREW: ${e.message}`);
} finally {
process.env.PATH = prev;
}
}
probe("normal PATH", process.env.PATH, "bash"); // "/bin/bash"
probe("starving which", bin, "bash"); // null after ~1000ms (SIGTERM folded away)
probe("starving which, absolute", bin, "/bin/bash"); // null — absolute path not exempt
await wrap("normal PATH (control)", process.env.PATH, undefined); // OK
await wrap("starving which", bin, undefined); // Shell 'bash' not found in PATH
await wrap("starving which, binShell=/bin/bash", bin, "/bin/bash"); // Shell '/bin/bash' not found in PATH
await wrap("no which on PATH", fix, undefined); // same message (ENOENT folded away)
```
Output on my machine (Node v26, pinned 0.0.73):
```
whichSync('bash') [normal PATH] 2ms -> /bin/bash
whichSync('bash') [starving which] 1001ms -> null
whichSync('/bin/bash') [starving which, absolute] 1002ms -> null
[normal PATH (control)] OK: bwrap --new-session --die-with-parent --bind / ...
[starving which] THREW: Shell 'bash' not found in PATH
[starving which, binShell=/bin/bash] THREW: Shell '/bin/bash' not found in PATH
[no which on PATH] THREW: Shell 'bash' not found in PATH
```
## Suggested fixes (any of these would help; the first is the real fix)
1. **Resolve in-process in the Node fallback** instead of forking `which` — walk `process.env.PATH` with `fs.statSync`/access checks, matching `Bun.which` semantics. This eliminates the fork, the 1s wall-clock deadline, and the entire class of "the lookup tool itself failed" errors. A per-process or TTL cache would also remove the per-exec fork cost on the happy path.
2. **Preserve the failure class**: return/throw `error.code`, `signal`, and `status` so embedders can distinguish "timed out" from "not installed" instead of getting a message that misstates the system.
3. **Skip the lookup for absolute `binShell`**: a `binShell` containing a path separator should be used directly (with an executability check), not routed through `which` — the current behavior reports `/bin/bash` as "not found in PATH", which contradicts itself.
Happy to turn the repro into a PR for (1) + (3) if the approach sounds right.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.