argoproj / argoproj/argo-workflows

Workflow controller occurs `fatal error: stack overflow` when specific named workflow is reconciled

Closed
#16,376 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

When a workflow with a specific combination of workflow name, step name, and referenced template name is reconciled, the workflow-controller crashes with `fatal error: stack overflow` in `childrenFulfilled` and `childrenFulfilledHelper`.
https://github.com/argoproj/argo-workflows/blob/v3.7.15/workflow/controller/operator.go#L2616-L2648

In our reproduction, the workflow stays in `Running` and the controller repeatedly restarts. This affects not only the problematic workflow but also the whole controller process.

The issue occurs with the following structure:

- Workflow name is `custom-job-thbh7`
- The workflow has a step named `custom-job`
- That step references a `WorkflowTemplate` or `ClusterWorkflowTemplate` template also named `custom-job`
- The referenced template is a `steps` template
- Inside that referenced template, the executed step is named `custom-job-main`
- That step references a pod/container template also named `custom-job-main`

We confirmed that the same problem occurs whether the referenced template is defined in a namespaced `WorkflowTemplate` or in a `ClusterWorkflowTemplate`.

In the actual incident, the workflow name was defined with `generateName`. The problem was discovered when a workflow was accidentally submitted with the generated name `custom-job-thbh7`. For the minimal reproduction below, we use a fixed `metadata.name: custom-job-thbh7` so that the same generated node IDs can be reproduced reliably.

With this combination, the workflow status contains a cyclic node relationship. For example:

```yaml
Nodes:
custom-job-thbh7:
Children:
custom-job-thbh7-1277661780

custom-job-thbh7-1277661780:
Boundary ID: custom-job-thbh7
Children:
custom-job-thbh7-3328663899

custom-job-thbh7-3328663899:
Boundary ID: custom-job-thbh7
Children:
custom-job-thbh7-3969105591

custom-job-thbh7-3969105591:
Boundary ID: custom-job-thbh7-3328663899
Children:
custom-job-thbh7-1277661780
```

The last node points back to `custom-job-thbh7-1277661780`, which is an ancestor node. After this cyclic node graph is created, the controller repeatedly recurses through the children while checking whether child nodes are fulfilled, and eventually crashes:

The Argo UI also shows an abnormal loop-like graph for this workflow. The `custom-job` node is connected in a way that visually returns to an earlier node instead of progressing to the pod node.
Image

```text
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow

github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2633
...
github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilled
/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2647
```

For comparison, if we submit the same workflow structure with a different workflow name, for example `custom-job-abcde`, the workflow succeeds normally. Its node graph is acyclic and includes the expected pod node:
Image

```yaml
custom-job-abcde:
Children:
custom-job-abcde-3032388424

custom-job-abcde-3032388424:
Children:
custom-job-abcde-3045544495

custom-job-abcde-3045544495:
Children:
custom-job-abcde-783001307

custom-job-abcde-783001307:
Children:
custom-job-abcde-2932551912

custom-job-abcde-2932551912:
Type: Pod
Phase: Succeeded
```

We also tested several variations to narrow down the condition. The issue did not occur in the following cases:

- The workflow name is changed from `custom-job-thbh7` to another value, such as `custom-job-abcde`.
- The workflow step does not reference another `steps` template, and instead directly references a pod/container template.
- The workflow step name or the referenced template name is changed from `custom-job` to another value.
- The inner step name or the pod/container template name is changed from `custom-job-main` to another value.
- Multiple inner steps are defined, but the executed, non-skipped step does not use the `custom-job-main` step/template name combination.

Therefore, the crash seems to require both the nested `steps` template structure and this specific set of generated node names, but it does not seem to depend on whether the referenced template is namespaced or cluster-scoped. This suggests that the controller can create an invalid cyclic node graph from otherwise valid workflow/template names, instead of simply rejecting the workflow or processing it normally.

We also checked the same problematic workflow on v3.6. In v3.6, submitting the workflow did not cause the workflow-controller to hit the fatal stack overflow, and the workflow completed. However, the node relationship was still cyclic in the same way. It looks like the invalid cyclic node graph can already be created, but the fatal crash is triggered by the `childrenFulfilled` / `childrenFulfilledHelper` logic that was added in v3.7.

Expected behavior:

- Reusing the same name for a step and the template it references should not create a cyclic node graph.
- The workflow should either run to completion or fail gracefully.
- The workflow-controller should not crash or enter a restart loop because of a malformed or cyclic node relationship.
- If such a cyclic node graph is detected, the controller should handle it defensively, for example by marking the workflow as failed with a clear error instead of recursing until stack overflow.

### Version(s)

v3.7.15, v4.0.6

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: custom-job-thbh7
labels:
workflows.argoproj.io/archive-strategy: "false"
spec:
arguments:
parameters:
- name: MESSAGE
entrypoint: main
templates:
- name: main
steps:
- - name: custom-job
templateRef:
name: workflow-template-debug
template: custom-job
arguments:
parameters:
- name: message
value: "{{workflow.parameters.MESSAGE}}"
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: workflow-template-debug
spec:
templates:
- name: custom-job
inputs:
parameters:
- name: message
steps:
- - name: custom-job-main
template: custom-job-main
arguments:
parameters:
- name: message
value: "{{inputs.parameters.message}}"
- name: custom-job-main
inputs:
parameters:
- name: message
container:
image: foo # in-house image based on rocky linux 9.

command: [echo]
args: ["{{inputs.parameters.message}}"]
resources:
limits:
memory: "1000Mi"
cpu: "1"
requests:
memory: "500Mi"
cpu: "500m"
```

### Logs from the workflow controller

```text
workflow-controller-65768587fc-jjldk workflow-controller time="2026-07-01T23:21:09.621Z" level=info msg="update pod event" pod=custom-job-thbh7-custom-job-main-1277661780
workflow-controller-65768587fc-jjldk workflow-controller time="2026-07-01T23:21:10.800Z" level=info msg="update pod event" pod=custom-job-thbh7-custom-job-main-1277661780
workflow-controller-65768587fc-jjldk workflow-controller time="2026-07-01T23:21:12.360Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2169762057 namespace=argo workflow=custom-job-thbh7
workflow-controller-65768587fc-jjldk workflow-controller time="2026-07-01T23:21:12.361Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=custom-job-thbh7
workflow-controller-65768587fc-jjldk workflow-controller runtime: goroutine stack exceeds 1000000000-byte limit
workflow-controller-65768587fc-jjldk workflow-controller runtime: sp=0xc02b1b04e0 stack=[0xc02b1b0000, 0xc04b1b0000]
workflow-controller-65768587fc-jjldk workflow-controller fatal error: stack overflow
workflow-controller-65768587fc-jjldk workflow-controller
workflow-controller-65768587fc-jjldk workflow-controller runtime stack:
workflow-controller-65768587fc-jjldk workflow-controller runtime.throw({0x2ff8174?, 0x435c9b?})
workflow-controller-65768587fc-jjldk workflow-controller /usr/local/go/src/runtime/panic.go:1094 +0x48 fp=0xc000195e98 sp=0xc000195e68 pc=0x4812e8
workflow-controller-65768587fc-jjldk workflow-controller runtime.newstack()
workflow-controller-65768587fc-jjldk workflow-controller /usr/local/go/src/runtime/stack.go:1159 +0x5bd fp=0xc000195fc8 sp=0xc000195e98 pc=0x46489d
workflow-controller-65768587fc-jjldk workflow-controller runtime.morestack()
workflow-controller-65768587fc-jjldk workflow-controller /usr/local/go/src/runtime/asm_amd64.s:620 +0x7d fp=0xc000195fd0 sp=0xc000195fc8 pc=0x4875bd
workflow-controller-65768587fc-jjldk workflow-controller
workflow-controller-65768587fc-jjldk workflow-controller goroutine 487 gp=0xc000f40fc0 m=4 mp=0xc000180008 [running]:
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/pkg/apis/workflow/v1alpha1.Nodes.Get(0xc0036c7b90, {0xc004e88f80, 0x1b})
workflow-controller-65768587fc-jjldk workflow-controller /go/src/github.com/argoproj/argo-workflows/pkg/apis/workflow/v1alpha1/workflow_types.go:1820 +0x1ca fp=0xc02b1b04f0 sp=0xc02b1b04e8 pc=0x1af250a
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper(0xc000bdfa40, 0xc016d92a80, 0xc04b1ab9d0)
workflow-controller-65768587fc-jjldk workflow-controller /go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2629 +0x1a5 fp=0xc02b1b07f8 sp=0xc02b1b04f0 pc=0x26ff205
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper(0xc000bdfa40, 0xc016d92900, 0xc04b1ab9d0)
workflow-controller-65768587fc-jjldk workflow-controller /go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2633 +0x1c5 fp=0xc02b1b0b00 sp=0xc02b1b07f8 pc=0x26ff225
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper(0xc000bdfa40, 0xc016d92780, 0xc04b1ab9d0)
workflow-controller-65768587fc-jjldk workflow-controller /go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2633 +0x1c5 fp=0xc02b1b0e08 sp=0xc02b1b0b00 pc=0x26ff225
...
workflow-controller-65768587fc-jjldk workflow-controller ...691732 frames elided...
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper(0xc000bdfa40, 0xc00573c600, 0xc04b1ab9d0)
workflow-controller-65768587fc-jjldk workflow-controller /go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2633 +0x1c5 fp=0xc04b1a4920 sp=0xc04b1a4618 pc=0x26ff225
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilledHelper(0xc000bdfa40, 0xc00573c480, 0xc04b1ab9d0)
workflow-controller-65768587fc-jjldk workflow-controller /go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2633 +0x1c5 fp=0xc04b1a4c28 sp=0xc04b1a4920 pc=0x26ff225
...
workflow-controller-65768587fc-jjldk workflow-controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).childrenFulfilled(...)
workflow-controller-65768587fc-jjldk workflow-controller /go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:2647
...
workflow-controller-65768587fc-jjldk workflow-controller runtime.goexit({})
workflow-controller-65768587fc-jjldk workflow-controller /usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000f03fe8 sp=0xc000f03fe0 pc=0x4892e1
workflow-controller-65768587fc-jjldk workflow-controller created by github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).Run in goroutine 108
workflow-controller-65768587fc-jjldk workflow-controller /go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:392 +0x19bc
workflow-controller-65768587fc-jjldk workflow-controller
workflow-controller-65768587fc-jjldk workflow-controller goroutine 515 gp=0xc000a56540 m=nil [sync.Cond.Wait, 8 minutes]:
workflow-controller-65768587fc-jjldk workflow-controller runtime.gopark(0xc005592ae0?, 0xc0009f6a80?, 0xe0?, 0xc9?, 0xc000690230?)
workflow-controller-65768587fc-jjldk workflow-controller /usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc008e53468 sp=0xc008e53448 pc=0x48140e
workflow-controller-65768587fc-jjldk workflow-controller runtime.goparkunlock(...)
workflow-controller-65768587fc-jjldk workflow-controller /usr/local/go/src/runtime/proc.go:466
workflow-controller-65768587fc-jjldk workflow-controller sync.runtime_notifyListWait(0xc00042bcd0, 0x2a582)
workflow-controller-65768587fc-jjldk workflow-controller /usr/local/go/src/runtime/sema.go:606 +0x159 fp=0xc008e534b8 sp=0xc008e53468 pc=0x482bf9
workflow-controller-65768587fc-jjldk workflow-controller sync.(*Cond).Wait(0x22f7c8f?)
workflow-controller-65768587fc-jjldk workflow-controller /usr/local/go/src/sync/cond.go:71 +0x73 fp=0xc008e534f0 sp=0xc008e534b8 pc=0x493293
workflow-controller-65768587fc-jjldk workflow-controller k8s.io/client-go/util/workqueue.(*Typed[...]).Get(0x35147a0)
workflow-controller-65768587fc-jjldk workflow-controller /go/pkg/mod/k8s.io/client-go@v0.33.1/util/workqueue/queue.go:277 +0x86 fp=0xc008e53558 sp=0xc008e534f0 pc=0x147a9c6
workflow-controller-65768587fc-jjldk workflow-controller k8s.io/client-go/util/workqueue.(*Typed[...]).Get()
workflow-controller-65768587fc-jjldk workflow-controller /go/pkg/mod/k8s.io/client-go@v0.33.1/util/workqueue/queue.go:273 +0x25 fp=0xc008e53578 sp=0xc008e53558 pc=0x147d7c5
workflow-controller-65768587fc-jjldk workflow-controller k8s.io/client-go/util/workqueue.(*delayingType[...]).Get()
workflow-controller-65768587fc-jjldk workflow-controller :1 +0x24 fp=0xc008e53590 sp=0xc008e53578 pc=0x147f3c4
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```

Contributor guide

Open the contributing guide

Research direction

Start in workflow/controller/operator.go at childrenFulfilled and childrenFulfilledHelper, then reproduce the cyclic node graph with the minimal Workflow and WorkflowTemplate YAML from the issue. Trace how the ancestor relationship is traversed and verify that the controller handles the cycle without stack overflow, crash, or restart loop, while preserving normal completion for the acyclic example.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.