coze-dev / coze-dev/coze-studio
工作流错误处理时引发 fatal error: concurrent map writes
- Dominant language
- TypeScript
- Stars
- 21.6k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
我在测试性能时,并发调用v1/workflwos/chat接口去执行工作流。工作流中有http节点访问外部服务,外部服务不可用从而触发了节点超时,随后服务崩溃。
看了下日志,原因是多个goroutine同时处理超时错误时,全局错误对象(NodeTimeoutErr、CancelErr)被所有goroutine共享,而 ChangeErrLevel 方法直接修改内部map导致了concurrent map writes
错误日志:
```
goroutine 910875 [running]:
internal/runtime/maps.fatal({0x402e2fd?, 0x40a8e5?})
runtime/panic.go:1046 +0x18
workflow-server/domain/workflow/entity/vo.(*wfErr).ChangeErrLevel(0xc034e829c0, {0x3fd9744, 0x4})
workflow-server/domain/workflow/entity/vo/node.go:151 +0x49
workflow-server/domain/workflow/internal/compose.(*nodeRunner[...]).onError(0xc043e9cf51, {0x47ccf70?, 0xc06b0279e0}, {0x4796c20, 0xc0378a9860})
workflow-server/domain/workflow/internal/compose/node_runner.go:848 +0x703
workflow-server/domain/workflow/internal/compose.(*nodeRunConfig[...]).invoke.func3.1()
workflow-server/domain/workflow/internal/compose/node_runner.go:319 +0x12c
workflow-server/domain/workflow/internal/compose.(*nodeRunConfig[...]).invoke.func3(0xc06acbe1e0, {0x0, 0x0, 0x0})
workflow-server/domain/workflow/internal/compose/node_runner.go:356 +0x3e6
github.com/cloudwego/eino/compose.(*runnablePacker[...]).Invoke(...)
github.com/cloudwego/eino@v0.4.8/compose/runnable.go:160
github.com/cloudwego/eino/compose.(*runnablePacker[...]).toComposableRunnable.func1({0x399ecc0, 0xc06acbe1e0}, {0x0?, 0xc062b4ab90?, 0xc03e3e04f8?})
github.com/cloudwego/eino@v0.4.8/compose/runnable.go:129 +0x1af
github.com/cloudwego/eino/compose.runnableInvoke({0x47ccf70?, 0xc06acbe2a0?}, 0xc043e9cf51?, {0x399ecc0?, 0xc06acbe1e0?}, {0x0?, 0xc044021008?, 0x23?})
github.com/cloudwego/eino@v0.4.8/compose/graph_run.go:100 +0x3f
github.com/cloudwego/eino/compose.(*taskManager).execute(0xc06753c5d0, 0xc0704db100)
github.com/cloudwego/eino@v0.4.8/compose/graph_manager.go:279 +0xbd
github.com/cloudwego/eino/compose.(*taskManager).submit(0xc06753c5d0, {0xc04c41b3a8, 0x1, 0x1})
github.com/cloudwego/eino@v0.4.8/compose/graph_manager.go:309 +0x285
github.com/cloudwego/eino/compose.(*runner).run(0xc043e46f00, {0x47ccf70, 0xc06753d380}, 0x0, {0x399ecc0, 0xc065c86990}, {0xc044021008, 0x23, 0x23})
github.com/cloudwego/eino@v0.4.8/compose/graph_run.go:293 +0x1bfa
github.com/cloudwego/eino/compose.(*runner).invoke(0x47c0ba0?, {0x47ccf70?, 0xc067147770?}, {0x399ecc0?, 0xc065c86990?}, {0xc044021008?, 0x3e22ac0?, 0x381db01?})
github.com/cloudwego/eino@v0.4.8/compose/graph_run.go:85 +0x45
github.com/cloudwego/eino/compose.(*graph).compile.(*runner).toComposableRunnable.func3({0x47ccf70, 0xc067147770}, {0x399ecc0, 0xc065c86990}, {0xc05d3fcd88?, 0x1?, 0x1?})
github.com/cloudwego/eino@v0.4.8/compose/graph_run.go:872 +0x85
github.com/cloudwego/eino/compose.toGenericRunnable[...].func1(0xc065c86990, {0xc043ee4008, 0x23, 0x0})
github.com/cloudwego/eino@v0.4.8/compose/runnable.go:405 +0x123
github.com/cloudwego/eino/compose.(*runnablePacker[...]).wrapRunnableCtx.func1(0xc065c86990?, {0xc043ee4008?, 0x23, 0x24})
github.com/cloudwego/eino@v0.4.8/compose/runnable.go:83 +0x6a
github.com/cloudwego/eino/compose.(*runnablePacker[...]).Invoke(...)
github.com/cloudwego/eino@v0.4.8/compose/runnable.go:160
workflow-server/domain/workflow/internal/compose.(*Workflow).AsyncRun.func2()
workflow-server/domain/workflow/internal/compose/workflow.go:169 +0x38
workflow-server/pkg/safego.Go.func1()
workflow-server/pkg/safego/safego.go:29 +0x53
created by workflow-server/pkg/safego.Go in goroutine 901093
workflow-server/pkg/safego/safego.go:26 +0x74
```
**To Reproduce**
Steps to reproduce the behavior:
1. 模拟并发测试
2. 执行调用外部http服务的节点
3. 外部http服务超时,导致节点超时
4. 多个goroutine会同时超时并进入错误处理流程
5. 服务因 fatal error: concurrent map writes 而崩溃
**Version:**
v0.5.0
v0.5.1貌似没看到有相关代码的修改
**Environment:**
```
AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/10355413/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/10355413/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1960900366=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/10355413/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/10355413/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/10355413/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.5'
GOWORK=''
PKG_CONFIG='pkg-config'
```
辛苦帮忙看下节点错误处理部分代码,以及其他使用了全局对象的代码段,是否有并发安全性问题,感谢
Contributor guide
Research direction
Start with domain/workflow/entity/vo/node.go at ChangeErrLevel, then trace the error path through domain/workflow/internal/compose/node_runner.go at onError and invoke. Reproduce concurrent HTTP-node timeouts against v0.5.0 and inspect other uses of the shared NodeTimeoutErr and CancelErr objects. Done means the workflow server no longer crashes with concurrent map writes under this scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100