argoproj / argoproj/argo-workflows

(Cron)Workflows fail silently if etcd leader changes during submission

Open
#14,880 3 comments 1 reaction 0 assignees View on GitHub
area/cron-workflows
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

When submitting a (Cron)Workflow, if an etcd leader change occurs at that exact time, the workflow submission fails silently. Argo does not retry the creation, and no visible indication is given to the user that the CronWorkflow run did not occur. This makes it very difficult to detect missing workflow executions without digging through controller logs.

**Expected Behavior:**
Argo should retry submission when transient errors like etcdserver: leader changed occur.
Alternatively, there should be a clear alert/event indicating the CronWorkflow execution failed.

**Actual Behavior:**
The (Cron)Workflow execution is skipped.
No retry attempt is made.

**Impact:**
Missed (Cron)Workflow executions go unnoticed.
Critical scheduled workflows may not run without operators realizing it.

### Version(s)

v3.7.2

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
*Steps to Reproduce:*
- Run an Argo (Cron)Workflow.
- Cause an etcd leader change during the (Cron)Workflow’s scheduled submission time (e.g., restart etcd leader).
- Observe that the workflow is not created.
- Check controller logs for the error message.
```

### Logs from the workflow controller

```text
time="2025-09-23T08:00:06.312Z" level=error msg="Failed to submit Workflow: [redacted] etcdserver: leader changed" conditionType=SubmissionError namespace=argo workflow=[redacted]
```

### Logs from in your workflow's wait container

```text
-
```

Contributor guide

Open the contributing guide

Research direction

Start at the workflow controller's CronWorkflow submission path and reproduce the etcd leader-change failure while checking the controller logs. Trace how the SubmissionError is handled, then define completion as retrying the transient submission or emitting a clear event or alert, with a regression test for the skipped execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.