argoproj / argoproj/argo-workflows
cron submit the same workflow twice
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
cron can submit the same workflow twice in an event of k8s apiserver outage
we can have the following timestamps
1. cron submitted workflow1
2. cron attempts to patch cronworkflow lastScheduleTime
3. k8s apiserver outage, unable to patch
4. controller restarts due health check fail/leader election fail
5. cron attemps to submit workflow unsuccessful but will keep trying since this [logic](https://github.com/argoproj/argo-workflows/blob/main/workflow/cron/operator.go#L131-L133) never updates the lastScheduleTime
6. workflow1 finished
7. cron submit the workflow1 with same name (due lastScheduleTime never updated, cron thinks we have a missing cron schedule)
Ideally, in this edge case (perhaps other edge case that trigger a similar situation), the cron should recover the lastScheduleTime and prevent a resubmission of the same workflow
https://github.com/argoproj/argo-workflows/blob/main/workflow/cron/operator.go#L131-L133
if workflow already exist we should patch lastScheduleTime using the existing workflow
### Version(s)
680ee6c
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```YAML
N/A
```
### Logs from the workflow controller
```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```
### Logs from in your workflow's wait container
```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```
Contributor guide
Research direction
Start in workflow/cron/operator.go at lines 131-133 and trace how a CronWorkflow submits a workflow and updates lastScheduleTime after an API-server outage or controller restart. Done means an already-created workflow lets the controller recover the schedule state and prevents the same scheduled workflow from being submitted twice; no specific test file is named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100