coze-dev / coze-dev/coze-studio

工作流错误处理时引发 fatal error: concurrent map writes

Open
#2,713 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
21.6k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

我在测试性能时,并发调用v1/workflwos/chat接口去执行工作流。工作流中有http节点访问外部服务,外部服务不可用从而触发了节点超时,随后服务崩溃。

看了下日志,原因是多个goroutine同时处理超时错误时,全局错误对象(NodeTimeoutErr、CancelErr)被所有goroutine共享,而 ChangeErrLevel 方法直接修改内部map导致了concurrent map writes

错误日志:
```
goroutine 910875 [running]:
internal/runtime/maps.fatal({0x402e2fd?, 0x40a8e5?})
runtime/panic.go:1046 +0x18
workflow-server/domain/workflow/entity/vo.(*wfErr).ChangeErrLevel(0xc034e829c0, {0x3fd9744, 0x4})
workflow-server/domain/workflow/entity/vo/node.go:151 +0x49
workflow-server/domain/workflow/internal/compose.(*nodeRunner[...]).onError(0xc043e9cf51, {0x47ccf70?, 0xc06b0279e0}, {0x4796c20, 0xc0378a9860})
workflow-server/domain/workflow/internal/compose/node_runner.go:848 +0x703
workflow-server/domain/workflow/internal/compose.(*nodeRunConfig[...]).invoke.func3.1()
workflow-server/domain/workflow/internal/compose/node_runner.go:319 +0x12c
workflow-server/domain/workflow/internal/compose.(*nodeRunConfig[...]).invoke.func3(0xc06acbe1e0, {0x0, 0x0, 0x0})
workflow-server/domain/workflow/internal/compose/node_runner.go:356 +0x3e6
github.com/cloudwego/eino/compose.(*runnablePacker[...]).Invoke(...)
github.com/cloudwego/eino@v0.4.8/compose/runnable.go:160
github.com/cloudwego/eino/compose.(*runnablePacker[...]).toComposableRunnable.func1({0x399ecc0, 0xc06acbe1e0}, {0x0?, 0xc062b4ab90?, 0xc03e3e04f8?})
github.com/cloudwego/eino@v0.4.8/compose/runnable.go:129 +0x1af
github.com/cloudwego/eino/compose.runnableInvoke({0x47ccf70?, 0xc06acbe2a0?}, 0xc043e9cf51?, {0x399ecc0?, 0xc06acbe1e0?}, {0x0?, 0xc044021008?, 0x23?})
github.com/cloudwego/eino@v0.4.8/compose/graph_run.go:100 +0x3f
github.com/cloudwego/eino/compose.(*taskManager).execute(0xc06753c5d0, 0xc0704db100)
github.com/cloudwego/eino@v0.4.8/compose/graph_manager.go:279 +0xbd
github.com/cloudwego/eino/compose.(*taskManager).submit(0xc06753c5d0, {0xc04c41b3a8, 0x1, 0x1})
github.com/cloudwego/eino@v0.4.8/compose/graph_manager.go:309 +0x285
github.com/cloudwego/eino/compose.(*runner).run(0xc043e46f00, {0x47ccf70, 0xc06753d380}, 0x0, {0x399ecc0, 0xc065c86990}, {0xc044021008, 0x23, 0x23})
github.com/cloudwego/eino@v0.4.8/compose/graph_run.go:293 +0x1bfa
github.com/cloudwego/eino/compose.(*runner).invoke(0x47c0ba0?, {0x47ccf70?, 0xc067147770?}, {0x399ecc0?, 0xc065c86990?}, {0xc044021008?, 0x3e22ac0?, 0x381db01?})
github.com/cloudwego/eino@v0.4.8/compose/graph_run.go:85 +0x45
github.com/cloudwego/eino/compose.(*graph).compile.(*runner).toComposableRunnable.func3({0x47ccf70, 0xc067147770}, {0x399ecc0, 0xc065c86990}, {0xc05d3fcd88?, 0x1?, 0x1?})
github.com/cloudwego/eino@v0.4.8/compose/graph_run.go:872 +0x85
github.com/cloudwego/eino/compose.toGenericRunnable[...].func1(0xc065c86990, {0xc043ee4008, 0x23, 0x0})
github.com/cloudwego/eino@v0.4.8/compose/runnable.go:405 +0x123
github.com/cloudwego/eino/compose.(*runnablePacker[...]).wrapRunnableCtx.func1(0xc065c86990?, {0xc043ee4008?, 0x23, 0x24})
github.com/cloudwego/eino@v0.4.8/compose/runnable.go:83 +0x6a
github.com/cloudwego/eino/compose.(*runnablePacker[...]).Invoke(...)
github.com/cloudwego/eino@v0.4.8/compose/runnable.go:160
workflow-server/domain/workflow/internal/compose.(*Workflow).AsyncRun.func2()
workflow-server/domain/workflow/internal/compose/workflow.go:169 +0x38
workflow-server/pkg/safego.Go.func1()
workflow-server/pkg/safego/safego.go:29 +0x53
created by workflow-server/pkg/safego.Go in goroutine 901093
workflow-server/pkg/safego/safego.go:26 +0x74
```

**To Reproduce**

Steps to reproduce the behavior:
1. 模拟并发测试
2. 执行调用外部http服务的节点
3. 外部http服务超时,导致节点超时
4. 多个goroutine会同时超时并进入错误处理流程
5. 服务因 fatal error: concurrent map writes 而崩溃

**Version:**

v0.5.0
v0.5.1貌似没看到有相关代码的修改

**Environment:**
```
AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/10355413/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/10355413/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1960900366=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/10355413/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/10355413/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/10355413/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.5'
GOWORK=''
PKG_CONFIG='pkg-config'
```

辛苦帮忙看下节点错误处理部分代码,以及其他使用了全局对象的代码段,是否有并发安全性问题,感谢

Contributor guide

Open the contributing guide

Research direction

Start with domain/workflow/entity/vo/node.go at ChangeErrLevel, then trace the error path through domain/workflow/internal/compose/node_runner.go at onError and invoke. Reproduce concurrent HTTP-node timeouts against v0.5.0 and inspect other uses of the shared NodeTimeoutErr and CancelErr objects. Done means the workflow server no longer crashes with concurrent map writes under this scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.