argoproj / argoproj/argo-workflows

Node offloading sometimes exposes unwanted stack traces from the database

Open
#14,905 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

Workflow 3.7.2

When enabling node offloading on a busy (high-load) controller, we see these stack traces

```
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller Stack:
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller fmt.(*pp).handleMethods@/usr/local/go/src/fmt/print.go:673
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller fmt.(*pp).printArg@/usr/local/go/src/fmt/print.go:756
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller fmt.(*pp).doPrint@/usr/local/go/src/fmt/print.go:1208
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller fmt.Append@/usr/local/go/src/fmt/print.go:289
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller log.(*Logger).Print.func1@/usr/local/go/src/log/log.go:261
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller log.(*Logger).output@/usr/local/go/src/log/log.go:238
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller log.(*Logger).Print@/usr/local/go/src/log/log.go:260
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/persist/sqldb.(*nodeOffloadRepo).Save@/go/src/github.com/argoproj/argo-workflows/persist/sqldb/offload_node_status_repo.go:89
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/hydrator.hydrator.Dehydrate.func1@/go/src/github.com/argoproj/argo-workflows/workflow/hydrator/hydrator.go:114
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/util/wait.Backoff.func1@/go/src/github.com/argoproj/argo-workflows/util/wait/backoff.go:15
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection@/go/pkg/mod/k8s.io/apimachinery@v0.33.1/pkg/util/wait/wait.go:150
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff@/go/pkg/mod/k8s.io/apimachinery@v0.33.1/pkg/util/wait/backoff.go:477
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/util/wait.Backoff@/go/src/github.com/argoproj/argo-workflows/util/wait/backoff.go:13
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/hydrator.hydrator.Dehydrate@/go/src/github.com/argoproj/argo-workflows/workflow/hydrator/hydrator.go:112
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).reapplyUpdate@/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:927
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).persistUpdates@/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:772
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate.func1@/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:191
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*wfOperationCtx).operate@/go/src/github.com/argoproj/argo-workflows/workflow/controller/operator.go:440
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller github.com/argoproj/argo-workflows/v3/workflow/controller.(*WorkflowController).processNextItem@/go/src/github.com/argoproj/argo-workflows/workflow/controller/controller.go:761
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller Error: ERROR: duplicate key value violates unique constraint "argo_workflows_pkey" (SQLSTATE 23505)
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller Time taken: 0.00145s
argowork-flows-argo-workflows-workflow-controller-5cd458cfl25rk controller Context: context.Background
```

From what we can tell, they can be ignored. But they are scary so we would like to not see them.

### Version(s)

v3.7.2

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
n/a
```

### Logs from the workflow controller

```text
as above
```

### Logs from in your workflow's wait container

```text
n/a
```

Contributor guide

Open the contributing guide

Research direction

Start at persist/sqldb/offload_node_status_repo.go:89 and trace the Save call through workflow/hydrator/hydrator.go:112-114, using the supplied duplicate-key stack trace as the symptom. Reproduce or inspect busy-controller node offloading, then verify that ignorable duplicate-key failures no longer emit unwanted stack traces while the error remains handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.