Massively parallel workflows causes panic

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

I launched a workflow with 10k individual steps using `withSequence`. Each container is simply calling a `sleep` command. This works, but if I launch a second such job concurrently, both fail with a panic:

```
&{0xc0000f6000 map[fromPhase:Error namespace:default toPhase:Failed workflow:danatest183641-d5mkl] 2022-11-18 19:22:11.887567952 +0000 UTC m=+11.333914714 panic workflow is already fulfilled }
```

### Version

v3.4.3

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
labels:
user: dana
name: danatest
namespace: default
spec:
entrypoint: flow
serviceAccountName: argo-workflows
templates:
- inputs: {}
metadata: {}
name: flow
outputs: {}
steps:
- - arguments:
parameters:
- name: line
value: '{{item}}'
name: dt-step
template: dt
withSequence:
end: 10001
start: 1
- container:
args:
- sleep 10
command:
- bash
- -c
image: ubuntu:focal
imagePullPolicy: IfNotPresent
name: ""
resources:
limits:
cpu: 10m
ephemeral-storage: 50Gi
memory: 100Mi
requests:
cpu: 10m
ephemeral-storage: 50Gi
memory: 100Mi
inputs:
parameters:
- name: line
metadata: {}
name: dt
outputs: {}
retryStrategy:
limit: 1
retryPolicy: OnError

```
```

### Logs from the workflow controller

```text
Due to the nature of this issue, there are a ton of logs... I'll relaunch using the simple workflow above and update the issue with the end of the logs:

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```

### Logs from in your workflow's wait container

```text
Due to the nature of this issue, there are a ton of logs:

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

time="2022-11-22T18:24:47.553Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-3612081654/main.log"
time="2022-11-22T18:24:47.553Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:24:47.565Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3612081654/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:24:47.642Z" level=info msg="Save artifact" artifactName=main-logs duration=89.617211ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3612081654/main.log
time="2022-11-22T18:24:47.642Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:24:47.642Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:24:47.655Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:24:47.656Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T18:24:47.656Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:24:47.656Z" level=info msg="Alloc=9966 TotalAlloc=16091 Sys=18642 NumGC=5 Goroutines=10"
time="2022-11-22T18:54:35.249Z" level=info msg="No output artifacts"
time="2022-11-22T18:54:35.249Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-4135713132/main.log"
time="2022-11-22T18:54:35.249Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:54:35.267Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-4135713132/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:54:35.351Z" level=info msg="Save artifact" artifactName=main-logs duration=101.983438ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-4135713132/main.log
time="2022-11-22T18:54:35.351Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:54:35.351Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:54:35.364Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:54:35.366Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:54:35.366Z" level=info msg="Alloc=6785 TotalAlloc=16311 Sys=22994 NumGC=6 Goroutines=12"
time="2022-11-22T18:50:55.811Z" level=info msg="No output parameters"
time="2022-11-22T18:50:55.811Z" level=info msg="No output artifacts"
time="2022-11-22T18:50:55.811Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-123733874/main.log"
time="2022-11-22T18:50:55.811Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:50:55.820Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-123733874/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:50:55.884Z" level=info msg="Save artifact" artifactName=main-logs duration=73.443331ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-123733874/main.log
time="2022-11-22T18:50:55.884Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:50:55.884Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:50:55.917Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:50:55.918Z" level=info msg="Alloc=7303 TotalAlloc=16309 Sys=22994 NumGC=5 Goroutines=11"
time="2022-11-22T19:21:35.241Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-1479836216/main.log"
time="2022-11-22T19:21:35.241Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:21:35.250Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1479836216/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:21:35.329Z" level=info msg="Save artifact" artifactName=main-logs duration=88.292794ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1479836216/main.log
time="2022-11-22T19:21:35.329Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:21:35.329Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:21:35.339Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:21:35.339Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T19:21:35.340Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T19:21:35.340Z" level=info msg="Alloc=7300 TotalAlloc=16172 Sys=19154 NumGC=5 Goroutines=11"
time="2022-11-22T17:58:38.085Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-1831161282/main.log"
time="2022-11-22T17:58:38.085Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T17:58:38.094Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1831161282/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T17:58:38.171Z" level=info msg="Save artifact" artifactName=main-logs duration=86.615362ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1831161282/main.log
time="2022-11-22T17:58:38.171Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T17:58:38.171Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T17:58:38.187Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T17:58:38.187Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T17:58:38.187Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T17:58:38.188Z" level=info msg="Alloc=10175 TotalAlloc=16274 Sys=22994 NumGC=4 Goroutines=9"
time="2022-11-22T18:47:13.921Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-2105457378/main.log"
time="2022-11-22T18:47:13.922Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:47:13.938Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2105457378/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:47:14.016Z" level=info msg="Save artifact" artifactName=main-logs duration=94.031747ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2105457378/main.log
time="2022-11-22T18:47:14.016Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:47:14.016Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:47:14.029Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:47:14.030Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T18:47:14.030Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:47:14.030Z" level=info msg="Alloc=10083 TotalAlloc=16266 Sys=22738 NumGC=5 Goroutines=10"
time="2022-11-22T19:05:59.462Z" level=info msg="No output artifacts"
time="2022-11-22T19:05:59.462Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-2246963646/main.log"
time="2022-11-22T19:05:59.462Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:05:59.473Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2246963646/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:05:59.554Z" level=info msg="Save artifact" artifactName=main-logs duration=91.399608ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2246963646/main.log
time="2022-11-22T19:05:59.554Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:05:59.554Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:05:59.567Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:05:59.568Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T19:05:59.568Z" level=info msg="Alloc=10026 TotalAlloc=16187 Sys=22482 NumGC=5 Goroutines=12"
time="2022-11-22T19:04:13.992Z" level=info msg="No output artifacts"
time="2022-11-22T19:04:13.992Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-3128008448/main.log"
time="2022-11-22T19:04:13.992Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:04:14.006Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3128008448/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:04:14.095Z" level=info msg="Save artifact" artifactName=main-logs duration=103.397546ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3128008448/main.log
time="2022-11-22T19:04:14.096Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:04:14.096Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:04:14.112Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:04:14.113Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T19:04:14.114Z" level=info msg="Alloc=6780 TotalAlloc=16205 Sys=22994 NumGC=6 Goroutines=11"
time="2022-11-22T18:46:58.917Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-254661592/main.log"
time="2022-11-22T18:46:58.917Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:46:58.930Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-254661592/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:46:58.996Z" level=info msg="Save artifact" artifactName=main-logs duration=78.801355ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-254661592/main.log
time="2022-11-22T18:46:58.996Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:46:58.996Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:46:59.008Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:46:59.009Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:46:59.009Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T18:46:59.009Z" level=info msg="Alloc=6854 TotalAlloc=16489 Sys=22738 NumGC=6 Goroutines=12"
time="2022-11-22T19:29:49.790Z" level=info msg="No output artifacts"
time="2022-11-22T19:29:49.790Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-2675014317/main.log"
time="2022-11-22T19:29:49.790Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:29:49.801Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2675014317/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:29:49.884Z" level=info msg="Save artifact" artifactName=main-logs duration=93.50396ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2675014317/main.log
time="2022-11-22T19:29:49.884Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:29:49.884Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:29:49.894Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:29:49.895Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T19:29:49.895Z" level=info msg="Alloc=6604 TotalAlloc=16223 Sys=22738 NumGC=5 Goroutines=11"
time="2022-11-22T18:14:03.877Z" level=info msg="Save artifact" artifactName=main-logs duration=109.224338ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1326185186/main.log
time="2022-11-22T18:14:03.877Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:14:03.877Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:14:03.893Z" level=info msg="Create workflowtaskresults 409"
time="2022-11-22T18:14:03.895Z" level=info msg="Patch workflowtaskresults 403"
time="2022-11-22T18:14:03.895Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io \"danatest173012-zdgxm-1326185186\" is forbidden: User \"system:serviceaccount:default:argo-workflows\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"default\""
time="2022-11-22T18:14:03.915Z" level=info msg="Patch pods 200"
time="2022-11-22T18:14:03.930Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T18:14:03.930Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:14:03.935Z" level=info msg="Alloc=10473 TotalAlloc=16587 Sys=22994 NumGC=5 Goroutines=9"
time="2022-11-22T19:31:01.556Z" level=info msg="No output artifacts"
time="2022-11-22T19:31:01.556Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-1668608042/main.log"
time="2022-11-22T19:31:01.556Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:31:01.569Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1668608042/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:31:01.652Z" level=info msg="Save artifact" artifactName=main-logs duration=95.175268ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1668608042/main.log
time="2022-11-22T19:31:01.652Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:31:01.652Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:31:01.665Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:31:01.666Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T19:31:01.666Z" level=info msg="Alloc=10057 TotalAlloc=16224 Sys=22994 NumGC=4 Goroutines=12"
time="2022-11-22T18:38:27.828Z" level=info msg="No output parameters"
time="2022-11-22T18:38:27.828Z" level=info msg="No output artifacts"
time="2022-11-22T18:38:27.829Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-184630328/main.log"
time="2022-11-22T18:38:27.829Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:38:27.842Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-184630328/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:38:27.924Z" level=info msg="Save artifact" artifactName=main-logs duration=95.097245ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-184630328/main.log
time="2022-11-22T18:38:27.924Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:38:27.924Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:38:27.937Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:38:27.938Z" level=info msg="Alloc=10006 TotalAlloc=16130 Sys=22994 NumGC=5 Goroutines=11"
time="2022-11-22T19:03:37.959Z" level=info msg="No output artifacts"
time="2022-11-22T19:03:37.959Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-1953733304/main.log"
time="2022-11-22T19:03:37.959Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:03:37.970Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1953733304/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:03:38.037Z" level=info msg="Save artifact" artifactName=main-logs duration=78.619821ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1953733304/main.log
time="2022-11-22T19:03:38.037Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:03:38.037Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:03:38.053Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:03:38.053Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T19:03:38.054Z" level=info msg="Alloc=10177 TotalAlloc=16441 Sys=22738 NumGC=5 Goroutines=11"
time="2022-11-22T18:29:13.666Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-2408006846/main.log"
time="2022-11-22T18:29:13.666Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:29:13.682Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2408006846/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:29:13.764Z" level=info msg="Save artifact" artifactName=main-logs duration=97.300114ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2408006846/main.log
time="2022-11-22T18:29:13.764Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:29:13.764Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:29:13.784Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:29:13.784Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:29:13.785Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T18:29:13.785Z" level=info msg="Alloc=10077 TotalAlloc=16280 Sys=22482 NumGC=5 Goroutines=9"
time="2022-11-22T19:30:31.830Z" level=info msg="No output artifacts"
time="2022-11-22T19:30:31.831Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-2523944742/main.log"
time="2022-11-22T19:30:31.831Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:30:31.842Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2523944742/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:30:31.921Z" level=info msg="Save artifact" artifactName=main-logs duration=90.616311ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2523944742/main.log
time="2022-11-22T19:30:31.922Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:30:31.922Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:30:31.936Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:30:31.937Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T19:30:31.937Z" level=info msg="Alloc=6791 TotalAlloc=16349 Sys=22738 NumGC=6 Goroutines=11"
time="2022-11-22T18:53:17.971Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-3815411058/main.log"
time="2022-11-22T18:53:17.971Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:53:17.981Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3815411058/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:53:18.048Z" level=info msg="Save artifact" artifactName=main-logs duration=77.046336ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3815411058/main.log
time="2022-11-22T18:53:18.048Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:53:18.048Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:53:18.060Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:53:18.061Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T18:53:18.061Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:53:18.061Z" level=info msg="Alloc=6632 TotalAlloc=16094 Sys=18386 NumGC=6 Goroutines=11"
time="2022-11-22T19:47:56.226Z" level=info msg="No output artifacts"
time="2022-11-22T19:47:56.226Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-3899782334/main.log"
time="2022-11-22T19:47:56.226Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:47:56.237Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3899782334/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:47:56.314Z" level=info msg="Save artifact" artifactName=main-logs duration=87.650729ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3899782334/main.log
time="2022-11-22T19:47:56.314Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:47:56.314Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:47:56.329Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:47:56.329Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T19:47:56.330Z" level=info msg="Alloc=6659 TotalAlloc=16165 Sys=18386 NumGC=6 Goroutines=11"
time="2022-11-22T19:45:23.531Z" level=info msg="Save artifact" artifactName=main-logs duration=156.252654ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3947189105/main.log
time="2022-11-22T19:45:23.532Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:45:23.532Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:45:23.545Z" level=info msg="Create workflowtaskresults 409"
time="2022-11-22T19:45:23.546Z" level=info msg="Patch workflowtaskresults 403"
time="2022-11-22T19:45:23.546Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io \"danatest173012-zdgxm-3947189105\" is forbidden: User \"system:serviceaccount:default:argo-workflows\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"default\""
time="2022-11-22T19:45:23.560Z" level=info msg="Patch pods 200"
time="2022-11-22T19:45:23.563Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T19:45:23.563Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T19:45:23.563Z" level=info msg="Alloc=9195 TotalAlloc=16886 Sys=22994 NumGC=5 Goroutines=11"
time="2022-11-22T19:05:14.764Z" level=info msg="No output artifacts"
time="2022-11-22T19:05:14.765Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-432451776/main.log"
time="2022-11-22T19:05:14.765Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:05:14.773Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-432451776/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:05:14.860Z" level=info msg="Save artifact" artifactName=main-logs duration=95.839696ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-432451776/main.log
time="2022-11-22T19:05:14.860Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:05:14.860Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:05:14.876Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:05:14.877Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T19:05:14.877Z" level=info msg="Alloc=10375 TotalAlloc=16274 Sys=22738 NumGC=4 Goroutines=11"
time="2022-11-22T17:42:39.308Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-3979284856/main.log"
time="2022-11-22T17:42:39.308Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T17:42:39.318Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3979284856/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T17:42:39.405Z" level=info msg="Save artifact" artifactName=main-logs duration=96.916487ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3979284856/main.log
time="2022-11-22T17:42:39.405Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T17:42:39.405Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T17:42:39.419Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T17:42:39.420Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T17:42:39.420Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T17:42:39.420Z" level=info msg="Alloc=6530 TotalAlloc=16094 Sys=22738 NumGC=5 Goroutines=10"
time="2022-11-22T18:24:29.801Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-4193285769/main.log"
time="2022-11-22T18:24:29.801Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:24:29.810Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-4193285769/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:24:29.893Z" level=info msg="Save artifact" artifactName=main-logs duration=92.110226ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-4193285769/main.log
time="2022-11-22T18:24:29.893Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:24:29.894Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:24:29.905Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:24:29.906Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:24:29.906Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T18:24:29.907Z" level=info msg="Alloc=10391 TotalAlloc=16271 Sys=22994 NumGC=4 Goroutines=10"
time="2022-11-22T18:13:43.413Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-1114560986/main.log"
time="2022-11-22T18:13:43.413Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:13:43.424Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1114560986/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:13:43.507Z" level=info msg="Save artifact" artifactName=main-logs duration=93.345277ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1114560986/main.log
time="2022-11-22T18:13:43.507Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:13:43.507Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:13:43.530Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:13:43.531Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T18:13:43.531Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:13:43.532Z" level=info msg="Alloc=6609 TotalAlloc=16237 Sys=22482 NumGC=5 Goroutines=9"
time="2022-11-22T19:46:17.260Z" level=info msg="No output artifacts"
time="2022-11-22T19:46:17.261Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-1229047627/main.log"
time="2022-11-22T19:46:17.261Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:46:17.274Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1229047627/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:46:17.358Z" level=info msg="Save artifact" artifactName=main-logs duration=97.159918ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1229047627/main.log
time="2022-11-22T19:46:17.358Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:46:17.358Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:46:17.370Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:46:17.372Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T19:46:17.372Z" level=info msg="Alloc=6790 TotalAlloc=16301 Sys=22994 NumGC=6 Goroutines=12"
time="2022-11-22T18:32:12.612Z" level=info msg="No output artifacts"
time="2022-11-22T18:32:12.612Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-1993611474/main.log"
time="2022-11-22T18:32:12.612Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:32:12.623Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1993611474/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:32:12.719Z" level=info msg="Save artifact" artifactName=main-logs duration=106.977348ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1993611474/main.log
time="2022-11-22T18:32:12.719Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:32:12.719Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:32:12.733Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:32:12.734Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T18:32:12.734Z" level=info msg="Alloc=6692 TotalAlloc=16243 Sys=22482 NumGC=6 Goroutines=11"
time="2022-11-22T19:07:03.688Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-2129512567/main.log"
time="2022-11-22T19:07:03.688Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:07:03.701Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2129512567/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:07:03.826Z" level=info msg="Save artifact" artifactName=main-logs duration=137.255702ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2129512567/main.log
time="2022-11-22T19:07:03.826Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:07:03.826Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:07:03.840Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:07:03.841Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T19:07:03.841Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T19:07:03.841Z" level=info msg="Alloc=6738 TotalAlloc=16218 Sys=22994 NumGC=6 Goroutines=9"
time="2022-11-22T20:02:44.296Z" level=info msg="No output parameters"
time="2022-11-22T20:02:44.296Z" level=info msg="No output artifacts"
time="2022-11-22T20:02:44.296Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-2651035276/main.log"
time="2022-11-22T20:02:44.296Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T20:02:44.308Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2651035276/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T20:02:44.381Z" level=info msg="Save artifact" artifactName=main-logs duration=84.962954ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2651035276/main.log
time="2022-11-22T20:02:44.381Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T20:02:44.381Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T20:02:44.402Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T20:02:44.407Z" level=info msg="Alloc=10029 TotalAlloc=16185 Sys=22738 NumGC=5 Goroutines=12"
time="2022-11-22T18:25:10.545Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:25:10.560Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3714305896/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:25:10.648Z" level=info msg="Save artifact" artifactName=main-logs duration=103.385139ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3714305896/main.log
time="2022-11-22T18:25:10.648Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:25:10.648Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:25:10.672Z" level=info msg="Create workflowtaskresults 409"
time="2022-11-22T18:25:10.673Z" level=info msg="Patch workflowtaskresults 403"
time="2022-11-22T18:25:10.673Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io \"danatest173012-zdgxm-3714305896\" is forbidden: User \"system:serviceaccount:default:argo-workflows\" cannot patch resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"default\""
time="2022-11-22T18:25:10.694Z" level=info msg="Patch pods 200"
time="2022-11-22T18:25:10.699Z" level=info msg="Alloc=7336 TotalAlloc=16821 Sys=22994 NumGC=6 Goroutines=12"
time="2022-11-22T19:06:52.066Z" level=info msg="No output parameters"
time="2022-11-22T19:06:52.066Z" level=info msg="No output artifacts"
time="2022-11-22T19:06:52.066Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-2930541696/main.log"
time="2022-11-22T19:06:52.066Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:06:52.079Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2930541696/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:06:52.153Z" level=info msg="Save artifact" artifactName=main-logs duration=86.864289ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-2930541696/main.log
time="2022-11-22T19:06:52.153Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:06:52.153Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:06:52.171Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:06:52.173Z" level=info msg="Alloc=6603 TotalAlloc=16282 Sys=22738 NumGC=5 Goroutines=12"
time="2022-11-22T18:00:26.853Z" level=info msg="No output artifacts"
time="2022-11-22T18:00:26.853Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-3310866260/main.log"
time="2022-11-22T18:00:26.853Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:00:26.863Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3310866260/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:00:26.934Z" level=info msg="Save artifact" artifactName=main-logs duration=81.478165ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-3310866260/main.log
time="2022-11-22T18:00:26.935Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:00:26.935Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:00:26.947Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:00:26.948Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:00:26.948Z" level=info msg="Alloc=8083 TotalAlloc=16220 Sys=18898 NumGC=5 Goroutines=11"
time="2022-11-22T18:25:11.767Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-1058990378/main.log"
time="2022-11-22T18:25:11.767Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:25:11.779Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1058990378/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:25:11.889Z" level=info msg="Save artifact" artifactName=main-logs duration=121.631894ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1058990378/main.log
time="2022-11-22T18:25:11.889Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:25:11.889Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:25:11.904Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:25:11.905Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:25:11.905Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T18:25:11.905Z" level=info msg="Alloc=6707 TotalAlloc=16153 Sys=22994 NumGC=6 Goroutines=9"
time="2022-11-22T19:07:00.919Z" level=info msg="No output parameters"
time="2022-11-22T19:07:00.919Z" level=info msg="No output artifacts"
time="2022-11-22T19:07:00.920Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-1157952236/main.log"
time="2022-11-22T19:07:00.920Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:07:00.935Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1157952236/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:07:01.037Z" level=info msg="Save artifact" artifactName=main-logs duration=116.875062ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1157952236/main.log
time="2022-11-22T19:07:01.037Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:07:01.037Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:07:01.052Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:07:01.054Z" level=info msg="Alloc=10170 TotalAlloc=16406 Sys=22994 NumGC=5 Goroutines=11"
time="2022-11-22T18:55:28.068Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-1246426068/main.log"
time="2022-11-22T18:55:28.068Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T18:55:28.078Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1246426068/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:55:28.147Z" level=info msg="Save artifact" artifactName=main-logs duration=79.860964ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1246426068/main.log
time="2022-11-22T18:55:28.147Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T18:55:28.147Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T18:55:28.160Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T18:55:28.161Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T18:55:28.161Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T18:55:28.161Z" level=info msg="Alloc=6616 TotalAlloc=16067 Sys=22482 NumGC=6 Goroutines=11"
time="2022-11-22T19:16:21.990Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-1434807874/main.log"
time="2022-11-22T19:16:21.990Z" level=info msg="Creating minio client using IAM role"
time="2022-11-22T19:16:22.009Z" level=info msg="Saving file to s3" bucket=reverie-prod-1-argo-artifacts endpoint=s3.amazonaws.com key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1434807874/main.log path=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:16:22.098Z" level=info msg="Save artifact" artifactName=main-logs duration=107.123772ms error="" key=danatest173012-zdgxm/danatest173012-zdgxm-dt-1434807874/main.log
time="2022-11-22T19:16:22.098Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2022-11-22T19:16:22.098Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2022-11-22T19:16:22.114Z" level=info msg="Create workflowtaskresults 201"
time="2022-11-22T19:16:22.115Z" level=info msg="Deadline monitor stopped"
time="2022-11-22T19:16:22.115Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2022-11-22T19:16:22.115Z" level=info msg="Alloc=6798 TotalAlloc=16438 Sys=22994 NumGC=6 Goroutines=9"
time="2022-11-22T18:32:09.855Z" level=info msg="No output parameters"
time="2022-11-22T18:32:09.855Z" level=info msg="No output artifacts"
time="2022-11-22T18:32:09.855Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: danatest173012-zdgxm/danatest173012-zdgxm-dt-183075524/main.log"
...
```
```

Reproduce the panic with the provided Kubernetes workflow using withSequence and two concurrent 10,000-step runs. Start by examining the workflow-controller path associated with the reported “workflow is already fulfilled” panic and use the controller logs to trace the failure. Done means concurrent massively parallel workflows complete without a panic.

Written by the indexing model from the issue text.