argoproj / argoproj/argo-workflows
(Cron)Workflows fail silently if etcd leader changes during submission
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
When submitting a (Cron)Workflow, if an etcd leader change occurs at that exact time, the workflow submission fails silently. Argo does not retry the creation, and no visible indication is given to the user that the CronWorkflow run did not occur. This makes it very difficult to detect missing workflow executions without digging through controller logs.
**Expected Behavior:**
Argo should retry submission when transient errors like etcdserver: leader changed occur.
Alternatively, there should be a clear alert/event indicating the CronWorkflow execution failed.
**Actual Behavior:**
The (Cron)Workflow execution is skipped.
No retry attempt is made.
**Impact:**
Missed (Cron)Workflow executions go unnoticed.
Critical scheduled workflows may not run without operators realizing it.
### Version(s)
v3.7.2
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```YAML
*Steps to Reproduce:*
- Run an Argo (Cron)Workflow.
- Cause an etcd leader change during the (Cron)Workflow’s scheduled submission time (e.g., restart etcd leader).
- Observe that the workflow is not created.
- Check controller logs for the error message.
```
### Logs from the workflow controller
```text
time="2025-09-23T08:00:06.312Z" level=error msg="Failed to submit Workflow: [redacted] etcdserver: leader changed" conditionType=SubmissionError namespace=argo workflow=[redacted]
```
### Logs from in your workflow's wait container
```text
-
```
Contributor guide
Research direction
Start at the workflow controller's CronWorkflow submission path and reproduce the etcd leader-change failure while checking the controller logs. Trace how the SubmissionError is handled, then define completion as retrying the transient submission or emitting a clear event or alert, with a regression test for the skipped execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100