LocalEnvironment.execute does not own the process tree: a timeout leaks forked descendants, and there is no way to cancel
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 205
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 92
Description
** Please make sure you read the contribution guide and file the issues in the
right place. **
[Contribution guide.](https://google.github.io/adk-docs/contributing-guide/)
**Describe the bug**
`LocalEnvironment.execute` does not own the lifecycle of the command it starts. Two consequences, which share one root cause:
1. **A timeout leaks forked descendants.** The command is spawned without `detached`, so it shares the parent's process group and there is no group to signal. The timeout path kills only the direct shell, and any process the command *forked* rather than *exec'd* survives, reparented and running.
2. **A command cannot be cancelled at all.** `execute` takes no `AbortSignal`, so a call made without `timeoutSeconds` is unescapable — the returned promise never settles and the process keeps running.
The timeout path currently reads:
```ts
// core/src/environment/local_environment.ts:139-148
timer = setTimeout(() => {
timedOut = true;
child.kill('SIGKILL');
// Killing the shell does not kill a command it forked rather than
// exec'd, and that survivor keeps the pipes open, which would hold
// 'close' back until it exits on its own. Release the read ends so
// the timeout is actually enforced.
child.stdout.destroy();
child.stderr.destroy();
}, timeoutSeconds * 1000);
```
That comment is accurate and the pipe-destroy fix is doing real work — it is what stops a surviving grandchild from holding `'close'` open indefinitely, and it is covered by a regression test (`core/test/environment/local_environment_test.ts`, "times out even when the command leaves a child holding the pipes open"). **This issue is not about that hang.** It is about the survivor itself: releasing the pipes makes the timeout *return*, but the orphan is still running, and nothing ever reaps it.
The class docstring already concedes the leak at `core/src/environment/local_environment.ts:71-74`:
> A timeout sends `SIGKILL` to the spawned shell; processes it forked itself may survive, and anything they write after the kill is not captured. On Windows such a survivor also keeps the working directory locked, so a `close` following a timeout can fail to remove a temporary workspace.
The Windows consequence noted there is worth restating: because the survivor holds the working directory, `close()` can fail to remove the temp workspace, so the leak escalates from a stray process to a leaked directory.
Relevant code:
- [`core/src/environment/local_environment.ts`](https://github.com/google/adk-js/blob/main/core/src/environment/local_environment.ts) — `spawn` at :125 (no `detached`), timeout branch at :139-148
- [`core/src/environment/base_environment.ts`](https://github.com/google/adk-js/blob/main/core/src/environment/base_environment.ts) — `abstract execute(command: string, timeoutSeconds?: number)` at :89-92 (no `AbortSignal`)
**To Reproduce**
Both cases as a single test file. Run on Linux with `npx vitest run `.
```ts
import {LocalEnvironment} from '@google/adk';
import {execSync} from 'node:child_process';
import {describe, expect, it} from 'vitest';
const NODE = `"${process.execPath}"`;
const MARKER = 'adk_orphan_probe';
const alive = () =>
Number(execSync(`pgrep -f ${MARKER} | wc -l`, {encoding: 'utf-8'}).trim());
describe('LocalEnvironment process lifecycle', () => {
it('leaks the grandchild after a timeout', async () => {
const env = new LocalEnvironment();
await env.initialize();
// A shell whose command forks a grandchild that outlives the deadline.
await env.writeFile(
'fork.cjs',
[
"const {spawn} = require('node:child_process');",
`spawn(process.execPath, ['-e', "process.title='${MARKER}'; setTimeout(()=>{}, 30000)"], {stdio: 'inherit'});`,
'setTimeout(() => {}, 30000);',
].join('\n'),
);
const result = await env.execute(`${NODE} fork.cjs`, 0.5);
await new Promise((r) => setTimeout(r, 300));
expect(result.timedOut).toBe(true);
expect(alive()).toBe(0); // FAILS: expected 2 to be +0
execSync(`pkill -9 -f ${MARKER} || true`);
await env.close();
}, 30_000);
it('cannot cancel a command that has no timeout', async () => {
const env = new LocalEnvironment();
await env.initialize();
await env.writeFile(
'hang.cjs',
[`process.title='${MARKER}';`, 'setTimeout(() => {}, 30000);'].join('\n'),
);
// There is no AbortSignal parameter, so there is no way to stop this.
const outcome = await Promise.race([
env.execute(`${NODE} hang.cjs`).then(() => 'completed'),
new Promise((r) => setTimeout(() => r('unescapable'), 2000)),
]);
expect(outcome).toBe('completed'); // FAILS: 'unescapable'
execSync(`pkill -9 -f ${MARKER} || true`);
await env.close();
}, 30_000);
});
```
Observed on `main` @ `10ae7bd`, Linux, Node v26.7.0:
```
timedOut=true survivors=2
outcome=unescapable
× leaks the grandchild after a timeout
→ expected 2 to be +0
× cannot cancel a command that has no timeout
→ expected 'unescapable' to be 'completed'
```
The timeout itself returns promptly (measured 509 ms for a 500 ms deadline), confirming the pipe-destroy fix works. The two survivors are the forked grandchild and its shell wrapper.
**Expected behavior**
- At the deadline, the command **and all of its descendants** are terminated and reaped, and `execute` returns `{timedOut: true}` promptly.
- `execute` accepts an `AbortSignal`; aborting terminates and reaps the same process tree, then rejects with an `AbortError`.
- Neither path leaves a process holding the working directory, so `close()` can always remove a temporary workspace.
**Desktop (please complete the following information):**
- OS: Linux / Node.js v26.7.0
- TS version/environment: TypeScript (repo main)
- ADK version: `@google/adk` main (`10ae7bd`)
**Additional context**
*Suggested fix.* Spawn each command in its own process group and signal the group:
```ts
const child = spawn(command, {shell: true, cwd, env, detached: true});
// …on timeout or abort:
process.kill(-child.pid!, 'SIGKILL');
```
One subtlety worth calling out for review: `detached: true` is not an optimization here, it is the **precondition**. Without it the child shares the parent's process group, and `process.kill(-pid)` would signal that group — killing the agent process itself. The two changes have to land together.
*Keep the existing pipe handling.* The `stdout.destroy()` / `stderr.destroy()` calls should stay even after the group kill, as defence against a descendant that has escaped the group (e.g. one that called `setsid` itself). Also worth preserving: the current implementation accumulates output via `'data'` listeners, so **output written before the deadline survives the timeout**. I verified this returns `EARLY_STDOUT_MARKER` correctly. Worth stating explicitly in any refactor, because it is easy to lose — the adk-python implementation re-reads with a second `communicate()` after the kill and consequently returns empty stdout on every timeout.
*Non-POSIX.* `detached` + `process.kill(-pid)` is POSIX-only. On Windows the options are, in descending order of correctness: a Job Object with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` (the only approach that reliably covers the whole tree); `taskkill /T /F /PID ` (pragmatic, and what most libraries use); `CREATE_NEW_PROCESS_GROUP` + `CTRL_BREAK_EVENT` (too weak to rely on, since a child may ignore it). Given the docstring already flags the Windows directory-locking consequence, `taskkill /T /F` would be a reasonable first step.
*Related.* adk-python's `LocalEnvironment` has the same missing process group (`environment/_local_environment.py`, `create_subprocess_shell` with no `start_new_session`, and a `proc.kill()` timeout branch), plus no cancellation cleanup and the output-loss problem described above. If the lifecycle contract is being fixed here it may be worth aligning both ports, since the intended semantics are shared.
Contributor guide
Assessment
This issue has not been assessed yet.