cloudwego / cloudwego/eino

adk: data race on shared reactConfig.cancelCtx when concurrent Run reuses one ChatModelAgent (tools/ReAct path)

Open
#1,177 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13k
Forks
1.1k
Avg merge
4h 6m
Merged PRs (30d)
41

Description

## Summary

`ChatModelAgent` that is **long-lived and reused across concurrent `Run` invocations** (tools/ReAct path enabled) races under `go test -race` on shared `reactConfig` / `modelWrapperConfig` fields that are mutated at the start of each run to inject that run’s cancel scope.

This is distinct from [#775](https://github.com/cloudwego/eino/pull/775) (concurrent `Compile` on a shared chain/graph). On `v0.9.13` (and still present on `main` / `v0.10.0-alpha.13` / `alpha/10` as of 2026-08), graph construction already happens **inside** the run closure, but the **config object that carries per-run cancel state is still shared** across runs.

## Production scenario

Multi-tenant request gateway:

- Boot once: one `ChatModelAgent` (stable model, tools, Handlers / Instruction base).
- Per HTTP/WebSocket turn: `Run(ctx, messages)` with an independent cancel tree (caller `context.Cancel` + ADK cancel options when used).
- Concurrent turns on the **same** agent instance are required for throughput (N tenants, one process).

Expectation after [#775](https://github.com/cloudwego/eino/pull/775): concurrent `Run` on one agent should be data-race free if each run builds its own chain/graph. Observed: **still races** when tools force the ReAct run function.

## Root cause (design intent vs. shared mutation)

### Why `msgConf.cancelCtx = cancelCtx` exists (legitimate)

Per-run cancel **must** reach:

1. **ReAct safe-points** — `newReact` snapshots `cancelCtx := config.cancelCtx` and the cancel-check nodes call `cancelCtx.shouldCancel()` / `CancelAfterChatModel` / `CancelAfterToolCalls` (`adk/react.go`, after capturing from config).
2. **Model wrapper stack** — `buildModelWrappers` copies `config.cancelContext` onto `typedStateModelWrapper` and into `TypedModelContext` for WrapModel / retry / failover (`adk/wrappers.go`).
3. **Graph interrupt wiring** — after `compose.WithGraphInterrupt`, the **per-run** `*cancelContext` receives `setGraphInterruptFunc` (that part already uses the run-local pointer from `typedRunParams`, not the shared conf).

So the assignment is **not dead code**. It is the inject point that binds **this** run’s cancel scope into graph + model construction. Removing it without a replacement would break cooperative cancel at those safe-points.

### Why mutating a shared conf is unsafe

`buildMessageReActRunFunc` builds **one** `msgConf` **outside** the returned closure (once per agent via `once.Do` → `buildRunFunc`). Every concurrent run then does:

```go
// adk/chatmodel.go (v0.9.13) — inside the Run closure
msgConf.cancelCtx = cancelCtx
if msgConf.modelWrapperConf != nil {
msgConf.modelWrapperConf.cancelContext = cancelCtx
}
g, err := newReact(ctx, msgConf)
```

| Location (v0.9.13) | What is shared |
|---|---|
| `adk/chatmodel.go` ~L1119–1140 | `msgConf *reactConfig` allocated once |
| `adk/chatmodel.go` ~L1145–1148 | concurrent write of `cancelCtx` / `modelWrapperConf.cancelContext` |
| `adk/chatmodel.go` ~L1369–1396 | `once.Do` freezes single `typedRunFunc` |
| Twin path | Agentic ReAct ~L1275–1278 same pattern on `agenticConf` |

Effects under concurrency (beyond the race detector):

- Two runs can interleave writes so run A’s `newReact` observes run B’s cancel pointer (or a torn intermediate).
- `modelWrapperConf` is a **shared pointer**: writing `cancelContext` races with any other run reading/building wrappers.
- Sequential reuse still “works” because each run overwrites before build; **only concurrent reentry fails the memory model**.

`withCancelContext(ctx, cancelCtx)` alone does not make the shared-field write safe; graph nodes and wrappers also consume the config-held cancel snapshot at build time.

## Reproduction (minimal)

Module: `github.com/cloudwego/eino@v0.9.13`
Command: `go test -race -count=20 -run TestConcurrentRunSharedAgentWithTools`

```go
package adk_test

import (
"context"
"sync"
"testing"

"github.com/cloudwego/eino/adk"
"github.com/cloudwego/eino/components/model"
"github.com/cloudwego/eino/components/tool"
"github.com/cloudwego/eino/compose"
"github.com/cloudwego/eino/schema"
)

// stubTC is a ToolCallingChatModel that returns one empty assistant message.
type stubTC struct{}

func (stubTC) WithTools([]*schema.ToolInfo) (model.ToolCallingChatModel, error) { return stubTC{}, nil }
func (stubTC) Generate(context.Context, []*schema.Message, ...model.Option) (*schema.Message, error) {
return schema.AssistantMessage("ok", nil), nil
}
func (stubTC) Stream(context.Context, []*schema.Message, ...model.Option) (*schema.StreamReader[*schema.Message], error) {
return schema.StreamReaderFromArray([]*schema.Message{schema.AssistantMessage("ok", nil)}), nil
}

// noopTool forces the ReAct (tools) run path — no-tools path does not hit this conf mutation.
type noopTool struct{}

func (noopTool) Info(context.Context) (*schema.ToolInfo, error) {
return &schema.ToolInfo{Name: "ping", Desc: "noop"}, nil
}
func (noopTool) InvokableRun(context.Context, string, ...tool.Option) (string, error) { return "pong", nil }

func TestConcurrentRunSharedAgentWithTools(t *testing.T) {
ctx := context.Background()
agent, err := adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{
Name: "shared", Description: "race repro", Instruction: "hi",
Model: stubTC{},
ToolsConfig: adk.ToolsConfig{
ToolsNodeConfig: compose.ToolsNodeConfig{
Tools: []tool.BaseTool{noopTool{}},
},
},
})
if err != nil {
t.Fatal(err)
}

var wg sync.WaitGroup
for i := 0; i < 2; i++ {
wg.Add(1)
go func() {
defer wg.Done()
iter := agent.Run(context.Background(), &adk.AgentInput{
Messages: []*schema.Message{schema.UserMessage("q")},
EnableStreaming: false,
})
for {
ev, ok := iter.Next()
if !ok {
return
}
if ev != nil && ev.Err != nil {
t.Errorf("run err: %v", ev.Err)
}
}
}()
}
wg.Wait()
}
```

**Observed (representative `-race` report):** concurrent write at `adk/chatmodel.go` in `buildMessageReActRunFunc`’s closure (~L1145 and ~L1147), both stacks originating from `ChatModelAgent.Run` → shared agent, two goroutines.

**Control:** sequential double `Run` on the same agent (no goroutines) is clean under `-race`.
**Control:** concurrent `Run` **without** tools (no-tools run func) did not hit this pair of write sites in our measurements.

## Related work (not a fix for this bug)

| Item | Relation |
|---|---|
| [#775](https://github.com/cloudwego/eino/pull/775) concurrent compile race | Fixed shared chain/graph; **left** shared `msgConf` cancel inject |
| Cancel-scope PRs (#1143, #1151, etc.) | Nested cancel **semantics**; not conf ownership under concurrent outer `Run` |
| [#1170](https://github.com/cloudwego/eino/pull/1170) callback handler slice race | Different subsystem (still open at time of writing) |

No public issue found that names this exact `msgConf.cancelCtx` concurrent write.

## Proposed fix (preserve cancel semantics; fix ownership)

**Do not drop** per-run cancel injection. **Do not** keep writing into the once-built shared conf.

Inside the run closure, **shallow-copy** `reactConfig` and `modelWrapperConfig` before assigning cancel fields, then pass the **local** conf to `newReact` / compile (same pattern as per-request graph in #775):

```go
// Inside buildMessageReActRunFunc's returned func:
mp := any(p).(*typedRunParams[*schema.Message])
cancelCtx := mp.cancelCtx
ctx = withCancelContext(ctx, cancelCtx)

// Per-run conf: stable fields shared by value; cancel pointers are run-local.
runConf := *msgConf
if msgConf.modelWrapperConf != nil {
mw := *msgConf.modelWrapperConf
mw.cancelContext = cancelCtx
runConf.modelWrapperConf = &mw
}
runConf.cancelCtx = cancelCtx

g, err := newReact(ctx, &runConf)
// ... Compile as today; setGraphInterruptFunc still on cancelCtx (already run-local)
```

Apply the same pattern to the **agentic** twin (`agenticConf` ~L1275–1278).

### Why this is low blast radius

- Sequential behavior: each run still injects its own cancel before graph build — same as today.
- Concurrent behavior: each run mutates only its stack-local conf copies.
- Shared immutable (for this path) fields (`model`, `handlers` slices as read-only, `agentName`, `maxIterations`, tools node config pointer if not mutated during build) remain shared by design; only **per-run cancel** stops living on the shared object.
- Residual caution: if any future code mutates `toolsConfig` / `toolInfos` in place during a run, that would need the same copy-on-write discipline — orthogonal to cancel inject.

### Verification suggested for a fix PR

1. The repro above under `-race` × N — must be clean.
2. Existing cancel suite (`CancelAfterChatModel` / `CancelAfterToolCalls` / nested cancel) — must stay green (proves injection still reaches safe-points).
3. Optional: two concurrent runs with distinct cancel trees — canceling A must not mark B’s safe-points (semantic isolation, not only race detector).

## Environment

- Module: `github.com/cloudwego/eino v0.9.13` (also checked present on `main`, `v0.10.0-alpha.13`, branch `alpha/10` raw `adk/chatmodel.go`)
- Go race detector: on
- OS: linux/amd64

Happy to turn this into a PR if the approach above matches maintainers’ intent for agent reentrancy.

Contributor guide

Open the contributing guide

Research direction

Start in adk/chatmodel.go at buildMessageReActRunFunc and inspect the agentic twin around its agenticConf setup, then run TestConcurrentRunSharedAgentWithTools with go test -race -count=20. Preserve the per-run cancel injection while ensuring both ReAct paths use run-local configuration copies; verify the repro is race-free and the existing cancellation tests remain green, including isolation between concurrent cancel trees.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.