argoproj / argoproj/argo-workflows
Node offloading sometimes exposes unwanted stack traces from the database
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
Workflow 3.7.2
When enabling node offloading on a busy (high-load) controller, we see these stack traces
```
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller Stack:
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller fmt.(*pp).handleMethods@/usr/local/go/src/fmt/print.go:673
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller fmt.(*pp).printArg@/usr/local/go/src/fmt/print.go:756
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller fmt.(*pp).doPrint@/usr/local/go/src/fmt/print.go:1208
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller fmt.Append@/usr/local/go/src/fmt/print.go:289
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller log.(*Logger).Print.func1@/usr/local/go/src/log/log.go:261
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller log.(*Logger).output@/usr/local/go/src/log/log.go:238
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller log.(*Logger).Print@/usr/local/go/src/log/log.go:260
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/persist/sqldb.(*nodeOffloadRepo).Save@/go/src/github.com/argoproj/argo-workflows/persist/sqldb/offload_node_status_repo.go:89
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/hydrator.hydrator.Dehydrate.func1@/go/src/github.com/argoproj/argo-workflows/workflow/hydrator/hydrator.go:114
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/util/wait.Backoff.func1@/go/src/github.com/argoproj/argo-workflows/util/wait/backoff.go:15
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection@/go/pkg/mod/k8s.io/apimachinery@v0.33.1/pkg/util/wait/wait.go:150
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff@/go/pkg/mod/k8s.io/apimachinery@v0.33.1/pkg/util/wait/backoff.go:477
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/util/wait.Backoff@/go/src/github.com/argoproj/argo-workflows/util/wait/backoff.go:13
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/hydrator.hydrator.Dehydrate@/go/src/github.com/argoproj/argo-workflows/workflow/hydrator/hydrator.go:112
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).reapplyUpdate@/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:927
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).persistUpdates@/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:772
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate.func1@/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:191
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate@/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:440
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem@/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:761
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller Error: ERROR: duplicate key value violates unique constraint "argo_workflows_pkey" (SQLSTATE 23505)
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller Time taken: 0.00145s
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller Context: context.Background
```
From what we can tell, they can be ignored. But they are scary so we would like to not see them.
### Version(s)
v3.7.2
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```YAML
n/a
```
### Logs from the workflow controller
```text
as above
```
### Logs from in your workflow's wait container
```text
n/a
```
Contributor guide
Research direction
Start at persist/sqldb/offload_node_status_repo.go:89 and trace the Save call through workflow/hydrator/hydrator.go:112-114, using the supplied duplicate-key stack trace as the symptom. Reproduce or inspect busy-controller node offloading, then verify that ignorable duplicate-key failures no longer emit unwanted stack traces while the error remains handled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100