adk: data race on shared reactConfig.cancelCtx when concurrent Run reuses one ChatModelAgent (tools/ReAct path)
- Dominant language
- Go
- Stars
- 13k
- Forks
- 1.1k
- Avg merge
- 4h 6m
- Merged PRs (30d)
- 41
Description
## Summary
`ChatModelAgent` that is **long-lived and reused across concurrent `Run` invocations** (tools/ReAct path enabled) races under `go test -race` on shared `reactConfig` / `modelWrapperConfig` fields that are mutated at the start of each run to inject that run’s cancel scope.
This is distinct from [#775](https://github.com/cloudwego/eino/pull/775) (concurrent `Compile` on a shared chain/graph). On `v0.9.13` (and still present on `main` / `v0.10.0-alpha.13` / `alpha/10` as of 2026-08), graph construction already happens **inside** the run closure, but the **config object that carries per-run cancel state is still shared** across runs.
## Production scenario
Multi-tenant request gateway:
- Boot once: one `ChatModelAgent` (stable model, tools, Handlers / Instruction base).
- Per HTTP/WebSocket turn: `Run(ctx, messages)` with an independent cancel tree (caller `context.Cancel` + ADK cancel options when used).
- Concurrent turns on the **same** agent instance are required for throughput (N tenants, one process).
Expectation after [#775](https://github.com/cloudwego/eino/pull/775): concurrent `Run` on one agent should be data-race free if each run builds its own chain/graph. Observed: **still races** when tools force the ReAct run function.
## Root cause (design intent vs. shared mutation)
### Why `msgConf.cancelCtx = cancelCtx` exists (legitimate)
Per-run cancel **must** reach:
1. **ReAct safe-points** — `newReact` snapshots `cancelCtx := config.cancelCtx` and the cancel-check nodes call `cancelCtx.shouldCancel()` / `CancelAfterChatModel` / `CancelAfterToolCalls` (`adk/react.go`, after capturing from config).
2. **Model wrapper stack** — `buildModelWrappers` copies `config.cancelContext` onto `typedStateModelWrapper` and into `TypedModelContext` for WrapModel / retry / failover (`adk/wrappers.go`).
3. **Graph interrupt wiring** — after `compose.WithGraphInterrupt`, the **per-run** `*cancelContext` receives `setGraphInterruptFunc` (that part already uses the run-local pointer from `typedRunParams`, not the shared conf).
So the assignment is **not dead code**. It is the inject point that binds **this** run’s cancel scope into graph + model construction. Removing it without a replacement would break cooperative cancel at those safe-points.
### Why mutating a shared conf is unsafe
`buildMessageReActRunFunc` builds **one** `msgConf` **outside** the returned closure (once per agent via `once.Do` → `buildRunFunc`). Every concurrent run then does:
```go
// adk/chatmodel.go (v0.9.13) — inside the Run closure
msgConf.cancelCtx = cancelCtx
if msgConf.modelWrapperConf != nil {
msgConf.modelWrapperConf.cancelContext = cancelCtx
}
g, err := newReact(ctx, msgConf)
```
| Location (v0.9.13) | What is shared |
|---|---|
| `adk/chatmodel.go` ~L1119–1140 | `msgConf *reactConfig` allocated once |
| `adk/chatmodel.go` ~L1145–1148 | concurrent write of `cancelCtx` / `modelWrapperConf.cancelContext` |
| `adk/chatmodel.go` ~L1369–1396 | `once.Do` freezes single `typedRunFunc` |
| Twin path | Agentic ReAct ~L1275–1278 same pattern on `agenticConf` |
Effects under concurrency (beyond the race detector):
- Two runs can interleave writes so run A’s `newReact` observes run B’s cancel pointer (or a torn intermediate).
- `modelWrapperConf` is a **shared pointer**: writing `cancelContext` races with any other run reading/building wrappers.
- Sequential reuse still “works” because each run overwrites before build; **only concurrent reentry fails the memory model**.
`withCancelContext(ctx, cancelCtx)` alone does not make the shared-field write safe; graph nodes and wrappers also consume the config-held cancel snapshot at build time.
## Reproduction (minimal)
Module: `github.com/cloudwego/eino@v0.9.13`
Command: `go test -race -count=20 -run TestConcurrentRunSharedAgentWithTools`
```go
package adk_test
import (
"context"
"sync"
"testing"
"github.com/cloudwego/eino/adk"
"github.com/cloudwego/eino/components/model"
"github.com/cloudwego/eino/components/tool"
"github.com/cloudwego/eino/compose"
"github.com/cloudwego/eino/schema"
)
// stubTC is a ToolCallingChatModel that returns one empty assistant message.
type stubTC struct{}
func (stubTC) WithTools([]*schema.ToolInfo) (model.ToolCallingChatModel, error) { return stubTC{}, nil }
func (stubTC) Generate(context.Context, []*schema.Message, ...model.Option) (*schema.Message, error) {
return schema.AssistantMessage("ok", nil), nil
}
func (stubTC) Stream(context.Context, []*schema.Message, ...model.Option) (*schema.StreamReader[*schema.Message], error) {
return schema.StreamReaderFromArray([]*schema.Message{schema.AssistantMessage("ok", nil)}), nil
}
// noopTool forces the ReAct (tools) run path — no-tools path does not hit this conf mutation.
type noopTool struct{}
func (noopTool) Info(context.Context) (*schema.ToolInfo, error) {
return &schema.ToolInfo{Name: "ping", Desc: "noop"}, nil
}
func (noopTool) InvokableRun(context.Context, string, ...tool.Option) (string, error) { return "pong", nil }
func TestConcurrentRunSharedAgentWithTools(t *testing.T) {
ctx := context.Background()
agent, err := adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{
Name: "shared", Description: "race repro", Instruction: "hi",
Model: stubTC{},
ToolsConfig: adk.ToolsConfig{
ToolsNodeConfig: compose.ToolsNodeConfig{
Tools: []tool.BaseTool{noopTool{}},
},
},
})
if err != nil {
t.Fatal(err)
}
var wg sync.WaitGroup
for i := 0; i < 2; i++ {
wg.Add(1)
go func() {
defer wg.Done()
iter := agent.Run(context.Background(), &adk.AgentInput{
Messages: []*schema.Message{schema.UserMessage("q")},
EnableStreaming: false,
})
for {
ev, ok := iter.Next()
if !ok {
return
}
if ev != nil && ev.Err != nil {
t.Errorf("run err: %v", ev.Err)
}
}
}()
}
wg.Wait()
}
```
**Observed (representative `-race` report):** concurrent write at `adk/chatmodel.go` in `buildMessageReActRunFunc`’s closure (~L1145 and ~L1147), both stacks originating from `ChatModelAgent.Run` → shared agent, two goroutines.
**Control:** sequential double `Run` on the same agent (no goroutines) is clean under `-race`.
**Control:** concurrent `Run` **without** tools (no-tools run func) did not hit this pair of write sites in our measurements.
## Related work (not a fix for this bug)
| Item | Relation |
|---|---|
| [#775](https://github.com/cloudwego/eino/pull/775) concurrent compile race | Fixed shared chain/graph; **left** shared `msgConf` cancel inject |
| Cancel-scope PRs (#1143, #1151, etc.) | Nested cancel **semantics**; not conf ownership under concurrent outer `Run` |
| [#1170](https://github.com/cloudwego/eino/pull/1170) callback handler slice race | Different subsystem (still open at time of writing) |
No public issue found that names this exact `msgConf.cancelCtx` concurrent write.
## Proposed fix (preserve cancel semantics; fix ownership)
**Do not drop** per-run cancel injection. **Do not** keep writing into the once-built shared conf.
Inside the run closure, **shallow-copy** `reactConfig` and `modelWrapperConfig` before assigning cancel fields, then pass the **local** conf to `newReact` / compile (same pattern as per-request graph in #775):
```go
// Inside buildMessageReActRunFunc's returned func:
mp := any(p).(*typedRunParams[*schema.Message])
cancelCtx := mp.cancelCtx
ctx = withCancelContext(ctx, cancelCtx)
// Per-run conf: stable fields shared by value; cancel pointers are run-local.
runConf := *msgConf
if msgConf.modelWrapperConf != nil {
mw := *msgConf.modelWrapperConf
mw.cancelContext = cancelCtx
runConf.modelWrapperConf = &mw
}
runConf.cancelCtx = cancelCtx
g, err := newReact(ctx, &runConf)
// ... Compile as today; setGraphInterruptFunc still on cancelCtx (already run-local)
```
Apply the same pattern to the **agentic** twin (`agenticConf` ~L1275–1278).
### Why this is low blast radius
- Sequential behavior: each run still injects its own cancel before graph build — same as today.
- Concurrent behavior: each run mutates only its stack-local conf copies.
- Shared immutable (for this path) fields (`model`, `handlers` slices as read-only, `agentName`, `maxIterations`, tools node config pointer if not mutated during build) remain shared by design; only **per-run cancel** stops living on the shared object.
- Residual caution: if any future code mutates `toolsConfig` / `toolInfos` in place during a run, that would need the same copy-on-write discipline — orthogonal to cancel inject.
### Verification suggested for a fix PR
1. The repro above under `-race` × N — must be clean.
2. Existing cancel suite (`CancelAfterChatModel` / `CancelAfterToolCalls` / nested cancel) — must stay green (proves injection still reaches safe-points).
3. Optional: two concurrent runs with distinct cancel trees — canceling A must not mark B’s safe-points (semantic isolation, not only race detector).
## Environment
- Module: `github.com/cloudwego/eino v0.9.13` (also checked present on `main`, `v0.10.0-alpha.13`, branch `alpha/10` raw `adk/chatmodel.go`)
- Go race detector: on
- OS: linux/amd64
Happy to turn this into a PR if the approach above matches maintainers’ intent for agent reentrancy.
Contributor guide
Research direction
Start in adk/chatmodel.go at buildMessageReActRunFunc and inspect the agentic twin around its agenticConf setup, then run TestConcurrentRunSharedAgentWithTools with go test -race -count=20. Preserve the per-run cancel injection while ensuring both ReAct paths use run-local configuration copies; verify the repro is race-free and the existing cancellation tests remain green, including isolation between concurrent cancel trees.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100