argoproj / argoproj/argo-workflows

cron submit the same workflow twice

Open
#14,719 1 comment 0 reactions 0 assignees View on GitHub
area/cron-workflows
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

cron can submit the same workflow twice in an event of k8s apiserver outage

we can have the following timestamps
1. cron submitted workflow1
2. cron attempts to patch cronworkflow lastScheduleTime
3. k8s apiserver outage, unable to patch
4. controller restarts due health check fail/leader election fail
5. cron attemps to submit workflow unsuccessful but will keep trying since this [logic](https://github.com/argoproj/argo-workflows/blob/main/workflow/cron/operator.go#L131-L133) never updates the lastScheduleTime
6. workflow1 finished
7. cron submit the workflow1 with same name (due lastScheduleTime never updated, cron thinks we have a missing cron schedule)

Ideally, in this edge case (perhaps other edge case that trigger a similar situation), the cron should recover the lastScheduleTime and prevent a resubmission of the same workflow

https://github.com/argoproj/argo-workflows/blob/main/workflow/cron/operator.go#L131-L133
if workflow already exist we should patch lastScheduleTime using the existing workflow

### Version(s)

680ee6c

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
N/A
```

### Logs from the workflow controller

```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```

Contributor guide

Open the contributing guide

Research direction

Start in workflow/cron/operator.go at lines 131-133 and trace how a CronWorkflow submits a workflow and updates lastScheduleTime after an API-server outage or controller restart. Done means an already-created workflow lets the controller recover the schedule state and prevents the same scheduled workflow from being submitted twice; no specific test file is named.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.