3.5.6 Logs flooded with `Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io`

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [X] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

After upgrading to 3.5.6 we can see in logs a lot of warning of

```
{"level":"warning","msg":"Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"workflow-6gndn\": the object has been modified; please apply your changes to the latest version and try again Conflict","namespace":"customer-workflows","time":"2024-05-29T11:47:22.010Z","workflow":"workflow-6gndn"}
```

The issue was separated from https://github.com/argoproj/argo-workflows/issues/13140

### Version

3.5.6

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
parallelism: 50
resourceRateLimit:
limit: 100
burst: 3
mainContainer:
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 0.3
memory: 256M
requests:
cpu: 0.1
memory: 128M
executor:
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 0.3
memory: 256M
requests:
cpu: 0.1
memory: 128M
artifactRepository:
archiveLogs: true
s3:
bucket: argo-wf
endpoint: s3.eu-west-1.amazonaws.com
keyFormat: "{{workflow.labels.customer-id}}/{{workflow.name}}/{{pod.name}}"
region: eu-west-1
persistence:
archive: true
nodeStatusOffLoad: true
connectionPool:
connMaxLifetime: 300s
postgresql:
host: argo-rds
port: 5432
database: argo_production
tableName: argo_workflows
userNameSecret:
key: username
name: db-secret
passwordSecret:
key: password
name: db-secret
```

### Logs from the workflow controller

```text
kubectl logs -n argo deploy/argo-workflows-workflow-controller -c controller | grep workflow-c6d2s

{"key":"customer-workflows/workflow-c6d2s","level":"info","msg":"Workflow processing has been postponed due to max parallelism limit","time":"2024-06-05T14:04:46.775Z"}
{"level":"info","msg":"Updated phase -\u003e Pending","namespace":"customer-workflows","time":"2024-06-05T14:04:46.775Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Updated message -\u003e Workflow processing has been postponed because too many workflows are already running","namespace":"customer-workflows","time":"2024-06-05T14:04:46.775Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"workflow-c6d2s\": the object has been modified; please apply your changes to the latest version and try again Conflict","namespace":"customer-workflows","time":"2024-06-05T14:04:46.787Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Re-applying updates on latest version and retrying update","namespace":"customer-workflows","time":"2024-06-05T14:04:46.787Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Update retry attempt 1 successful","namespace":"customer-workflows","time":"2024-06-05T14:04:46.800Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Pending","resourceVersion":"245969123","time":"2024-06-05T14:04:46.800Z","workflow":"workflow-c6d2s"}
{"key":"customer-workflows/workflow-c6d2s","level":"info","msg":"Workflow processing has been postponed due to max parallelism limit","time":"2024-06-05T14:04:51.785Z"}
{"Phase":"Pending","ResourceVersion":"245969123","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:11:46.200Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":0,"time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Updated phase Pending -\u003e Running","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Updated message Workflow processing has been postponed because too many workflows are already running -\u003e ","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"DAG node workflow-c6d2s initialized Running","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:46.206Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:46.206Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-500067944, taskName task-02b979a3-e102-4e26-87ef-91ffe5c799a2","time":"2024-06-05T14:11:46.206Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-500067944, taskName task-02b979a3-e102-4e26-87ef-91ffe5c799a2","time":"2024-06-05T14:11:46.206Z"}
{"level":"info","msg":"All of node workflow-c6d2s.task-02b979a3-e102-4e26-87ef-91ffe5c799a2 dependencies [] completed","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Pod node workflow-c6d2s-500067944 initialized Pending","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Created pod: workflow-c6d2s.task-02b979a3-e102-4e26-87ef-91ffe5c799a2 (workflow-c6d2s-action-version-7142-task-02b979a3-e102-4e26-87ef-91ffe5c799a2-500067944)","namespace":"customer-workflows","time":"2024-06-05T14:11:46.237Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:46.237Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:46.237Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:46.237Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:46.237Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:11:46.237Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:11:46.237Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245995160","time":"2024-06-05T14:11:46.277Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245995286","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:11:51.238Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":1,"time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-c6d2s-500067944","old.message":"PodInitializing","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:51.240Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245995473","time":"2024-06-05T14:11:51.291Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245995601","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:11:56.286Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":1,"time":"2024-06-05T14:11:56.287Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:11:56.287Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-c6d2s-500067944","old.message":"","old.phase":"Running","old.progress":"0/1","time":"2024-06-05T14:11:56.287Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:56.287Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:56.288Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:56.290Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:56.290Z"}
{"level":"info","msg":"All of node workflow-c6d2s.task-15f3d4f5-0e50-4327-83cf-1923b124169a dependencies [task-02b979a3-e102-4e26-87ef-91ffe5c799a2] completed","namespace":"customer-workflows","time":"2024-06-05T14:11:56.290Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"customer-workflows","time":"2024-06-05T14:11:56.290Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Pod node workflow-c6d2s-1995060186 initialized Pending","namespace":"customer-workflows","time":"2024-06-05T14:11:56.290Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Created pod: workflow-c6d2s.task-15f3d4f5-0e50-4327-83cf-1923b124169a (workflow-c6d2s-action-version-7917-task-15f3d4f5-0e50-4327-83cf-1923b124169a-1995060186)","namespace":"customer-workflows","time":"2024-06-05T14:11:56.318Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:56.318Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:56.318Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:11:56.318Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:11:56.318Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245995803","time":"2024-06-05T14:11:56.339Z","workflow":"workflow-c6d2s"}
{"action":"deletePod","key":"customer-workflows/workflow-c6d2s-action-version-7142-task-02b979a3-e102-4e26-87ef-91ffe5c799a2-500067944/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:11:58.341Z"}
{"Phase":"Running","ResourceVersion":"245995932","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:01.317Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":2,"time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-c6d2s-1995060186","old.message":"PodInitializing","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:12:01.319Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:12:01.320Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:12:01.320Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:01.320Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:01.320Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245996166","time":"2024-06-05T14:12:01.333Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245996323","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:06.337Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":2,"time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node unchanged","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"PodInitializing","new.phase":"Pending","new.progress":"0/1","nodeID":"workflow-c6d2s-4280457137","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:06.339Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:06.339Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245996490","time":"2024-06-05T14:12:06.370Z","workflow":"workflow-c6d2s"}
{"action":"deletePod","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-15f3d4f5-0e50-4327-83cf-1923b124169a-1995060186/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:08.372Z"}
{"Phase":"Running","ResourceVersion":"245996679","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:11.369Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":3,"time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node unchanged","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:11.372Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:11.372Z","workflow":"workflow-c6d2s"}
{"action":"terminateContainers","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-ec0efb20-7a0f-452f-996e-e8b86bd723ff-4280457137/terminateContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:11.377Z"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245996843","time":"2024-06-05T14:12:11.384Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245997016","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:16.386Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":3,"time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node unchanged","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"PodInitializing","new.phase":"Pending","new.progress":"0/1","nodeID":"workflow-c6d2s-2361177406","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:16.388Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:16.388Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"customer-workflows","time":"2024-06-05T14:12:16.388Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245997196","time":"2024-06-05T14:12:16.405Z","workflow":"workflow-c6d2s"}
{"action":"deletePod","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-ec0efb20-7a0f-452f-996e-e8b86bd723ff-4280457137/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:18.411Z"}
{"Phase":"Running","ResourceVersion":"245997390","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:21.403Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":4,"time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-2361177406","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-c6d2s-2361177406","old.message":"","old.phase":"Running","old.progress":"0/1","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"customer-workflows","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"action":"terminateContainers","key":"customer-workflows/workflow-c6d2s-exit-handler-2361177406/terminateContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:21.410Z"}
{"level":"info","msg":"https://172.20.0.1:443/api/v1/namespaces/customer-workflows/pods/workflow-c6d2s-exit-handler-2361177406/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec\u0026command=kill\u0026command=15\u0026command=1\u0026container=wait\u0026stderr=true\u0026stdout=true\u0026tty=false","time":"2024-06-05T14:12:21.410Z"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245997561","time":"2024-06-05T14:12:21.423Z","workflow":"workflow-c6d2s"}
{"container":"wait","error":"unable to upgrade connection: container not found (\"wait\")","level":"info","msg":"signaled container","namespace":"customer-workflows","pod":"workflow-c6d2s-exit-handler-2361177406","stderr":null,"stdout":null,"time":"2024-06-05T14:12:21.438Z"}
{"level":"info","msg":"archiving workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:25.441Z","uid":"59086eb4-e239-4e99-ac40-91a4a90f5632","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Queueing Succeeded workflow customer-workflows/workflow-c6d2s for delete in 5m0s due to TTL","time":"2024-06-05T14:12:25.496Z"}
{"action":"killContainers","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-ec0efb20-7a0f-452f-996e-e8b86bd723ff-4280457137/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:41.377Z"}
{"action":"killContainers","key":"customer-workflows/workflow-c6d2s-exit-handler-2361177406/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:51.439Z"}
{"level":"info","msg":"Queueing Succeeded workflow customer-workflows/workflow-c6d2s for delete in 2m29s due to TTL","time":"2024-06-05T14:14:56.080Z"}
{"level":"info","msg":"Deleting garbage collected workflow 'customer-workflows/workflow-c6d2s'","time":"2024-06-05T14:17:26.000Z"}
{"level":"info","msg":"Workflow already deleted 'customer-workflows/workflow-c6d2s'","time":"2024-06-05T14:17:26.014Z"}
```

### Logs from in your workflow's wait container

```text
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
```

Start with the workflow controller deployment entry point and trace the workflow update and retry messages shown in the logs, focusing on the Kubernetes workflow resource conflict. Reproduce the behavior using the supplied workflow and controller configuration; done should mean successful workflow updates without the reported flood of conflict warnings.

Written by the indexing model from the issue text.