anthropics / anthropics/anthropic-sdk-go

Self-hosted worker: bash tool call never returns when the command uses setsid — session parked at requires_action forever (third wedge class, distinct from #377/#388)

Offen
#390 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Go
Sterne
1.2k
Forks
213
Ø Merge
1 T. 12 Std.
Gemergte PRs (30 T.)
11

Beschreibung

## Summary

A `bash` tool call whose command backgrounds a process with **`setsid`** can cause the tool dispatch to never complete: no `user.tool_result` is ever posted (not even a timeout error), the session goes `idle` with `stop_reason: requires_action` referencing the already-dispatched `tool_use`, and stays parked there indefinitely. Both nominal 120 s bounds (the bash tool's internal exec timer and the runner's `ToolTimeout`) fail to produce a result event.

This is a third, distinct session-wedge class alongside #377 (empty tool output → 400) and #388 (ask-ordering): the failing call had `evaluated_permission: "allow"` stamped on its `agent.tool_use` event, and its command explicitly ended with `echo OK`, so neither existing issue applies.

## Versions

- `ant` v1.16.0 (latest release), self-hosted worker via `ant beta:worker poll --workdir /workspace`
- pins `anthropic-sdk-go` v1.56.0; the relevant code is unchanged on `main`
- Container: `node:22-slim` base, K8s pod, PTY available

## Observed

The agent had made three prior attempts to background a long-running Node script (plain `timeout 280` foreground; plain `nohup … &`; then cleanup/pkill of strays) — all completed and posted results normally. The fourth attempt added `setsid`:

```bash
rm -f /tmp/aggregate-output.txt /tmp/aggregate-stderr.txt /tmp/aggregate.pid
cd /workspace/.worktrees/nexus/
setsid nohup node /workspace/skills//aggregate-lint.mjs > /tmp/aggregate-output.txt 2>/tmp/aggregate-stderr.txt < /dev/null &
disown
echo $! > /tmp/aggregate.pid
sleep 3
cat /tmp/aggregate.pid
ps -p "$(cat /tmp/aggregate.pid)" -o pid,cmd
echo OK
```

Event stream tail (from `GET /v1/sessions/{id}/events`):

1. `agent.tool_use` (bash, `evaluated_permission: "allow"`) — `2026-07-13T03:10:44.347Z`
2. `session.thread_status_idle` + `session.status_idle`, both `stop_reason: {type: "requires_action", event_ids: []}` — `03:10:44.524Z`
3. **Nothing else, ever.** No `user.tool_result`, no error result, no timeout message, for the ~1h51m until the pod died for unrelated reasons.

The orphaned `setsid` process itself kept running unsupervised after the wedge (in our case it eventually filled the node's ephemeral storage).

## Root-cause hypothesis (source trace)

In `tools/agenttoolset/bash.go`:

- `BashSession.exec` and the runner's `ToolTimeout` both fire fine as timers. The problem is the cleanup path they funnel into: on any Exec error, `bashTool.run` calls `t.restart()` → `BashSession.Close()`, which after `killProcessGroup(s.cmd.Process)` calls **`s.cmd.Wait()` unconditionally, with no timeout and no context** (`Close` in bash.go).
- `setsid` makes the backgrounded child a **session leader with no controlling terminal**. Per POSIX semantics, such a process can acquire the next tty it opens as its controlling terminal — and Node/CLI tools routinely probe `/dev/tty`. It can therefore re-attach to the very PTY slave the persistent bash session is using.
- If that interaction leaves the original `/bin/bash` (or the PTY teardown) in an uninterruptible state, SIGKILL is not promptly reaped, `cmd.Wait()` blocks forever, and the per-dispatch goroutine — the one responsible for building and POSTing the tool_result — never returns. `drainInFlight`'s 30 s drain gives up and logs, but no result event is ever sent for the dispatched tool_use, so the session waits for a `user.tool_result` that will never come.

I could not confirm the exact kernel-level wait state (the pod was evicted before inspection), but the event-stream evidence plus the unbounded `Wait()` is sufficient to reproduce the shape: any command that prevents the bash process from being reaped wedges the session permanently.

## Suggested fixes

1. **Bound the reap**: in `BashSession.Close()`, wait for `cmd.Wait()` in a goroutine with a deadline (e.g. 10 s); on timeout, log, abandon the process, and return — a leaked zombie is strictly better than a permanently parked session.
2. **Guarantee a result event per dispatch**: in the session tool runner, enforce the tool deadline at the dispatch level too — if `Execute` has not returned by `ToolTimeout` + grace, post an `is_error` tool_result ("tool execution abandoned after timeout") from the dispatch loop rather than depending on `Execute` returning. This makes *every* future hang-class bug self-healing at the protocol level.
3. Optionally: start the persistent shell with `setsid`-resistant PTY handling (e.g. open the PTY with `O_NOCTTY` where applicable) or document that `setsid` inside the bash tool is unsupported.

## Impact

Same operational blast radius as #377/#388: a self-hosted environment session parked forever with the worker looking healthy (liveness ≠ progress), requiring out-of-band detection and manual archive. Coding agents reach for `setsid`/`nohup` backgrounding on their own when a task exceeds the 120 s tool bound, so this fires in practice without any adversarial input.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.