Custom counters double-incremented when retryStrategy is specified

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

When a workflow template has retryStrategy configured (either via templateDefaults at workflow/controller level or on individual templates), custom Prometheus metrics are emitted twice for each task execution.

curl -s http://localhost:8080/metrics | grep workflow_counter

HELP argo_workflows_workflow_counter Counter should increment by 1

TYPE argo_workflows_workflow_counter counter

argo_workflows_workflow_counter{status="Succeeded",workflow_name="test-double-metrics-rhk9m"} 2

This leads to incorrect observability data. Without retryStrategy, the counter is incremented only once.

Expected behavior:

• Counter incremented by 1

Observed behavior:

• Counter incremented by 2

I suspect Both Retry wrapper and child node emit metrics independently, causing this issue. The solution is probably to skip metric emission for child nodes when they have a Retry parent node (in both handleNodeFulfilled and executeTemplate in operator.go).

Version(s)

v3.7.3

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-double-metrics-
  namespace: default
spec:
  entrypoint: main-dag
  templateDefaults:
    retryStrategy:
      limit: "2"
      retryPolicy: Always
  templates:
  - name: main-dag
    dag:
      tasks:
      - name: task-a
        template: simple-task
    metrics:
      prometheus:
      - name: workflow_counter
        help: "Counter should increment by 1"
        labels:
        - key: workflow_name
          value: "{{workflow.name}}"
        - key: status
          value: "{{status}}"
        counter:
          value: "1"
  
  - name: simple-task
    container:
      image: busybox
      command: [sh, -c]
      args: ["echo 'Task completed'; exit 0"]

Logs from the workflow controller

kubectl logs argo-workflows-workflow-controller-c47f66f46-9mp8w | grep test-double-metrics-rhk9m

time=2026-02-06T20:31:26.950Z level=INFO msg="Processing workflow" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform ResourceVersion=902542 Phase=""
time=2026-02-06T20:31:26.954Z level=INFO msg="resolved artifact repository" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform artifactRepositoryRef=default-artifact-repository
time=2026-02-06T20:31:26.954Z level=INFO msg="Task-result reconciliation" namespace=vmsp-platform numObjs=0 component=workflow_worker workflow=test-double-metrics-rhk9m
time=2026-02-06T20:31:26.954Z level=INFO msg="updated phase" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform toPhase=Running fromPhase=""
time=2026-02-06T20:31:26.954Z level=WARN msg="Node was nil, will be initialized as type Skipped" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform
time=2026-02-06T20:31:26.954Z level=INFO msg="Event occurred" reason=WorkflowRunning message="Workflow Running" component=workflow_worker object.name=test-double-metrics-rhk9m object.namespace=vmsp-platform fieldPath="" kind=Workflow apiVersion=argoproj.io/v1alpha1 type=Normal
time=2026-02-06T20:31:26.954Z level=INFO msg="was unable to obtain node, letting display name to be nodeName" boundaryID="" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform
time=2026-02-06T20:31:26.954Z level=INFO msg="node initialized" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform node=test-double-metrics-rhk9m phase=Running message=""
time=2026-02-06T20:31:26.955Z level=INFO msg="was unable to obtain node, letting display name to be nodeName" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform boundaryID=""
time=2026-02-06T20:31:26.955Z level=INFO msg="node initialized" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform message="" node=test-double-metrics-rhk9m-217547824 phase=Running
time=2026-02-06T20:31:26.955Z level=WARN msg="was unable to obtain the node" workflow=test-double-metrics-rhk9m namespace=vmsp-platform component=workflow_worker nodeID=test-double-metrics-rhk9m-269037917 taskName=task-a
time=2026-02-06T20:31:26.955Z level=WARN msg="was unable to obtain the node" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform nodeID=test-double-metrics-rhk9m-269037917 taskName=task-a
time=2026-02-06T20:31:26.955Z level=WARN msg="was unable to obtain the node" workflow=test-double-metrics-rhk9m namespace=vmsp-platform component=workflow_worker nodeID=test-double-metrics-rhk9m-269037917 taskName=task-a
time=2026-02-06T20:31:26.955Z level=INFO msg="All of node dependencies completed" nodeName=test-double-metrics-rhk9m(0).task-a dependencies=[] component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform
time=2026-02-06T20:31:26.955Z level=WARN msg="Node was nil, will be initialized as type Skipped" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform
time=2026-02-06T20:31:26.955Z level=INFO msg="node initialized" namespace=vmsp-platform phase=Running message="" node=test-double-metrics-rhk9m-269037917 component=workflow_worker workflow=test-double-metrics-rhk9m
time=2026-02-06T20:31:26.955Z level=INFO msg="node initialized" message="" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform node=test-double-metrics-rhk9m-2621229548 phase=Pending
time=2026-02-06T20:31:26.955Z level=INFO msg=getExecutorLogOpts nodeID=test-double-metrics-rhk9m-2621229548 loglevel=info component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform nodeName=test-double-metrics-rhk9m(0).task-a(0)
time=2026-02-06T20:31:26.955Z level=INFO msg=getExecutorLogOpts workflow=test-double-metrics-rhk9m namespace=vmsp-platform loglevel=info nodeName=test-double-metrics-rhk9m(0).task-a(0) nodeID=test-double-metrics-rhk9m-2621229548 component=workflow_worker
time=2026-02-06T20:31:26.956Z level=INFO msg=getExecutorLogOpts loglevel=info component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform nodeName=test-double-metrics-rhk9m(0).task-a(0) nodeID=test-double-metrics-rhk9m-2621229548
time=2026-02-06T20:31:26.967Z level=INFO msg="add pod event" component=pod_controller pod=test-double-metrics-rhk9m-simple-task-2621229548
time=2026-02-06T20:31:26.967Z level=INFO msg="Created pod" nodeID=test-double-metrics-rhk9m-2621229548 component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform nodeName=test-double-metrics-rhk9m(0).task-a(0) podName=test-double-metrics-rhk9m-simple-task-2621229548
time=2026-02-06T20:31:26.967Z level=INFO msg="TaskSet Reconciliation" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform
time=2026-02-06T20:31:26.967Z level=INFO msg=reconcileAgentPod workflow=test-double-metrics-rhk9m namespace=vmsp-platform component=workflow_worker
time=2026-02-06T20:31:26.968Z level=INFO msg="Workflow to be dehydrated" namespace=vmsp-platform "Workflow Size"=2351 component=workflow_worker workflow=test-double-metrics-rhk9m
time=2026-02-06T20:31:26.974Z level=INFO msg="update pod event" pod=test-double-metrics-rhk9m-simple-task-2621229548 component=pod_controller
time=2026-02-06T20:31:26.976Z level=INFO msg="Workflow update successful" resourceVersion=902546 phase=Running component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform
time=2026-02-06T20:31:26.977Z level=INFO msg="Event occurred" object.name=test-double-metrics-rhk9m object.namespace=vmsp-platform fieldPath="" kind=Workflow apiVersion=argoproj.io/v1alpha1 component=workflow_worker type=Normal reason=WorkflowNodeRunning message="Running node test-double-metrics-rhk9m"
time=2026-02-06T20:31:26.977Z level=INFO msg="Event occurred" apiVersion=argoproj.io/v1alpha1 component=workflow_worker type=Normal reason=WorkflowNodeRunning message="Running node test-double-metrics-rhk9m(0)" object.name=test-double-metrics-rhk9m object.namespace=vmsp-platform fieldPath="" kind=Workflow
time=2026-02-06T20:31:26.977Z level=INFO msg="Event occurred" apiVersion=argoproj.io/v1alpha1 type=Normal reason=WorkflowNodeRunning message="Running node test-double-metrics-rhk9m(0).task-a" object.name=test-double-metrics-rhk9m object.namespace=vmsp-platform fieldPath="" component=workflow_worker kind=Workflow
time=2026-02-06T20:31:27.001Z level=INFO msg="update pod event" component=pod_controller pod=test-double-metrics-rhk9m-simple-task-2621229548
time=2026-02-06T20:31:28.610Z level=INFO msg="update pod event" component=pod_controller pod=test-double-metrics-rhk9m-simple-task-2621229548
time=2026-02-06T20:31:29.622Z level=INFO msg="update pod event" component=pod_controller pod=test-double-metrics-rhk9m-simple-task-2621229548
time=2026-02-06T20:31:30.633Z level=INFO msg="update pod event" component=pod_controller pod=test-double-metrics-rhk9m-simple-task-2621229548
time=2026-02-06T20:31:31.643Z level=INFO msg="update pod event" component=pod_controller pod=test-double-metrics-rhk9m-simple-task-2621229548
time=2026-02-06T20:31:32.926Z level=INFO msg="update pod event" component=pod_controller pod=test-double-metrics-rhk9m-simple-task-2621229548
time=2026-02-06T20:31:36.969Z level=INFO msg="Processing workflow" workflow=test-double-metrics-rhk9m namespace=vmsp-platform Phase=Running ResourceVersion=902546 component=workflow_worker
time=2026-02-06T20:31:36.969Z level=INFO msg="Task-result reconciliation" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform numObjs=1
time=2026-02-06T20:31:36.970Z level=INFO msg="node phase changed" workflow=test-double-metrics-rhk9m namespace=vmsp-platform component=workflow_worker toPhase=Succeeded node=test-double-metrics-rhk9m-269037917 fromPhase=Running
time=2026-02-06T20:31:36.970Z level=INFO msg="node finished" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform node=test-double-metrics-rhk9m-269037917 finishedAt=2026-02-06T20:31:36.970779101Z
time=2026-02-06T20:31:36.970Z level=INFO msg="Outbound nodes set" workflow=test-double-metrics-rhk9m namespace=vmsp-platform nodeID=test-double-metrics-rhk9m-217547824 outbound=[test-double-metrics-rhk9m-2621229548] component=workflow_worker
time=2026-02-06T20:31:36.970Z level=INFO msg="node phase changed" workflow=test-double-metrics-rhk9m namespace=vmsp-platform node=test-double-metrics-rhk9m-217547824 fromPhase=Running toPhase=Succeeded component=workflow_worker
time=2026-02-06T20:31:36.970Z level=INFO msg="node finished" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform node=test-double-metrics-rhk9m-217547824 finishedAt=2026-02-06T20:31:36.970935372Z
time=2026-02-06T20:31:36.971Z level=INFO msg="node phase changed" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform toPhase=Succeeded node=test-double-metrics-rhk9m fromPhase=Running
time=2026-02-06T20:31:36.971Z level=INFO msg="node finished" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform node=test-double-metrics-rhk9m finishedAt=2026-02-06T20:31:36.971223512Z
time=2026-02-06T20:31:36.971Z level=INFO msg="TaskSet Reconciliation" namespace=vmsp-platform component=workflow_worker workflow=test-double-metrics-rhk9m
time=2026-02-06T20:31:36.971Z level=INFO msg=reconcileAgentPod component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform
time=2026-02-06T20:31:36.971Z level=INFO msg="updated phase" namespace=vmsp-platform fromPhase=Running toPhase=Succeeded component=workflow_worker workflow=test-double-metrics-rhk9m
time=2026-02-06T20:31:36.971Z level=INFO msg="Marking workflow completed" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform
time=2026-02-06T20:31:36.971Z level=INFO msg="Event occurred" apiVersion=argoproj.io/v1alpha1 type=Normal reason=WorkflowSucceeded message="Workflow completed" object.name=test-double-metrics-rhk9m object.namespace=vmsp-platform fieldPath="" component=workflow_worker kind=Workflow
time=2026-02-06T20:31:36.972Z level=INFO msg="Workflow to be dehydrated" workflow=test-double-metrics-rhk9m namespace=vmsp-platform component=workflow_worker "Workflow Size"=2996
time=2026-02-06T20:31:36.982Z level=INFO msg="Workflow update successful" component=workflow_worker workflow=test-double-metrics-rhk9m namespace=vmsp-platform resourceVersion=902624 phase=Succeeded
time=2026-02-06T20:31:36.982Z level=INFO msg="Event occurred" reason=WorkflowNodeSucceeded message="Succeeded node test-double-metrics-rhk9m" object.name=test-double-metrics-rhk9m object.namespace=vmsp-platform fieldPath="" component=workflow_worker kind=Workflow apiVersion=argoproj.io/v1alpha1 type=Normal
time=2026-02-06T20:31:36.982Z level=INFO msg="Event occurred" message="Succeeded node test-double-metrics-rhk9m(0)" component=workflow_worker object.name=test-double-metrics-rhk9m object.namespace=vmsp-platform fieldPath="" kind=Workflow apiVersion=argoproj.io/v1alpha1 type=Normal reason=WorkflowNodeSucceeded
time=2026-02-06T20:31:36.982Z level=INFO msg="Event occurred" kind=Workflow apiVersion=argoproj.io/v1alpha1 component=workflow_worker type=Normal reason=WorkflowNodeRunning message="Running node test-double-metrics-rhk9m(0).task-a(0)" object.name=test-double-metrics-rhk9m object.namespace=vmsp-platform fieldPath=""
time=2026-02-06T20:31:36.982Z level=INFO msg="Event occurred" apiVersion=argoproj.io/v1alpha1 type=Normal reason=WorkflowNodeSucceeded component=workflow_worker message="Succeeded node test-double-metrics-rhk9m(0).task-a(0)" object.name=test-double-metrics-rhk9m object.namespace=vmsp-platform fieldPath="" kind=Workflow
time=2026-02-06T20:31:36.982Z level=INFO msg="Event occurred" fieldPath="" kind=Workflow apiVersion=argoproj.io/v1alpha1 type=Normal reason=WorkflowNodeSucceeded message="Succeeded node test-double-metrics-rhk9m(0).task-a" component=workflow_worker object.name=test-double-metrics-rhk9m object.namespace=vmsp-platform
time=2026-02-06T20:31:36.982Z level=INFO msg="Queueing workflow for delete due to TTL" component=gc_controller phase=Succeeded workflow=vmsp-platform/test-double-metrics-rhk9m addAfter=1h0m0s
time=2026-02-06T20:31:37.987Z level=INFO msg="queueing pod for cleanup after" podName=test-double-metrics-rhk9m-simple-task-2621229548 action=deletePod after=8h0m0s namespace=vmsp-platform component=pod_controller

Logs from in your workflow's wait container

kubectl logs -n vmsp-platform -c wait -l workflows.argoproj.io/workflow=test-double-metrics-rhk9m,workflow.argoproj.io/phase!=Succeeded

time="2026-02-06T20:31:28.826Z" level=info msg="Starting Workflow Executor" version=v0.0.0+unknown
time="2026-02-06T20:31:28.829Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2026-02-06T20:31:28.829Z" level=info msg="Executor initialized" deadline="2026-02-07 02:31:26 +0000 UTC" includeScriptOutput=false namespace=vmsp-platform podName=test-double-metrics-rhk9m-simple-task-2621229548 templateName=simple-task version="&Version{Version:v0.0.0+unknown,BuildDate:1970-01-01T00:00:00Z,GitCommit:,GitTag:,GitTreeState:,GoVersion:go1.24.11 X:boringcrypto,Compiler:gc,Platform:linux/amd64,}"
time="2026-02-06T20:31:28.841Z" level=info msg="Starting deadline monitor"
time="2026-02-06T20:31:30.842Z" level=info msg="Main container completed" error="<nil>"
time="2026-02-06T20:31:30.842Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2026-02-06T20:31:30.842Z" level=info msg="No output parameters"
time="2026-02-06T20:31:30.842Z" level=info msg="No output artifacts"
time="2026-02-06T20:31:30.853Z" level=info msg="Alloc=10606 TotalAlloc=15670 Sys=24149 NumGC=4 Goroutines=8"

Start in operator.go, focusing on handleNodeFulfilled and executeTemplate, which the report identifies as metric-emission paths. Run the minimal workflow with templateDefaults.retryStrategy and inspect the controller's /metrics output. Done means the custom workflow_counter reports one increment for the successful task, both with and without retryStrategy.