panic: send on closed channel in typedAgentTool.InvokableRun (adk/agent_tool.go:243) after cancellation — path missed by #929
- Dominant language
- Go
- Stars
- 13k
- Forks
- 1.1k
- Avg merge
- 4h 6m
- Merged PRs (30d)
- 41
Description
## Summary
`panic: send on closed channel` crashes the whole process when a context is
cancelled while an in-flight agent-as-tool call (`typedAgentTool`, e.g. the
`task` tool of a deep agent) is still running. When the sub-agent finishes
after the parent layer has already closed its event generator, the event
forwarding loop in `typedAgentTool.InvokableRun` calls `gen.Send(event)` on a
closed `AsyncGenerator` and panics.
PR #929 fixed the same panic shape for the `chatModelAgentExecCtx.send` path
(`isImmediateCancelled()` fast path + `trySend()` safety net), but the
`agent_tool.go` event forwarding path was not covered by that fix.
## Environment
- eino v0.9.13 (goroutine stack below)
- Also checked v0.9.19 (latest stable) and current `main`: `adk/agent_tool.go:243`
still calls `gen.Send(event)` and `internal/channel.go:42` still panics on
closed — unchanged.
## Stack trace
```
panic: send on closed channel
goroutine 810 [running]:
github.com/cloudwego/eino/internal.(*UnboundedChan[...]).Send(...)
.../eino@v0.9.13/internal/channel.go:42
github.com/cloudwego/eino/adk.(*AsyncGenerator[...]).Send(...)
.../eino@v0.9.13/adk/utils.go:44
github.com/cloudwego/eino/adk.(*typedAgentTool[...]).InvokableRun(...)
.../eino@v0.9.13/adk/agent_tool.go:243 +0x9c8
github.com/cloudwego/eino/adk/prebuilt/deep.(*typedTaskTool[...]).InvokableRun(...)
.../eino@v0.9.13/adk/prebuilt/deep/task_tool.go:174 +0x1dc
github.com/cloudwego/eino/compose.(*invokableToolWithCallback).InvokableRun(...)
.../eino@v0.9.13/compose/tool_node.go:661 +0x8c
... (compose tool_node / adk wrappers middleware chain) ...
```
## Timeline
Observed in a production deep-agent orchestration (3 layers: supervisor →
sector commander → workers, all connected via agent-as-tool). Cancelling the
run while task delegations are in flight:
```
Goroutine A (run owner) Goroutine B (in-flight tool call)
──────────────────────────── ─────────────────────────────────
ctx cancelled (user stop)
graph converges, logs:
[GraphRunError] context has been canceled
parent layer generator closed
sub-agent iteration returns events
typedAgentTool.InvokableRun loop:
gen.Send(event) → PANIC
```
The log line immediately before the panic is the graph-level cancellation
error, i.e. the run had already converged on the consumer side — the producer
side (agent_tool event forwarding) then hit the closed channel ~milliseconds
later. The wider the fan-out of concurrent agent-as-tool calls, the easier
this race is to hit on stop/cancel (in our case: several in-flight task
delegations at once).
## Root cause
1. `internal/channel.go:42` — `UnboundedChan.Send` panics when closed
(`TrySend` in the same file already returns `false` instead).
2. `adk/agent_tool.go:243` — the event forwarding loop of
`typedAgentTool.InvokableRun` uses the panicking `gen.Send(event)`.
Notably, `AsyncGenerator.trySend()` already exists
(`adk/utils.go:47`, added by #929) and is exactly the safe variant needed
here — it is just not used on this path.
## Suggested fix
In `adk/agent_tool.go:243`, replace the bare send with the safe variant and
drop the event when the generator is already closed (the consumer is gone by
definition in this race):
```go
if gen != nil {
if event.Action == nil || event.Action.Interrupted == nil {
// ...
if !gen.trySend(event) {
break // generator closed after cancellation: no consumer left
}
}
}
```
(or apply the same two-layer guard style as #929 if a fast-path cancellation
check is preferred.)
## Workaround
For anyone hitting this before a fix lands: wrapping tool calls with a
`compose.ToolMiddleware` that `defer recover()`s and converts the panic into
a `ctx.Err()`-wrapped tool error keeps the process alive, since the middleware
sits on the call stack above `typedAgentTool.InvokableRun`.
Contributor guide
Research direction
Start in adk/agent_tool.go around line 243 and compare its event-forwarding loop with the safe sending path in adk/utils.go and the cancellation fix from #929. Reproduce or add coverage for cancellation during an in-flight agent-as-tool call, then verify the closed generator causes no panic and the run completes with its cancellation error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ai-infra-agents
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100