ArtifactGC with both OnCompletion and OnDeletion as well as forceFinalizerRemoval dont work

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

When running a workflow with two artifacts, one being configured to be garbage collected `OnCompletion` and one to be garbage collected `OnDeletion`, argo will only garbage collect the one with `OnCompletion`. This only happens when also `artifactGC.forceFinalizerRemoval` is set to `true`.

From the logs i can that after the `OnCompletion` artifactGC task finishes, the finalizer `workflows.argoproj.io/artifact-gc` gets already removed, even though an artifact with `OnDeletion` strategy is still present.
After deleting the workflow, argo will not cleanup this artifact as expected.

### Version(s)

v3.6.5

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: test-argo-artifact-gc2
namespace: argo
spec:
artifactGC:
forceFinalizerRemoval: true
entrypoint: output-artifact-gc
templates:
- container:
args:
- echo "hello" > /tmp/hello.txt
command:
- sh
- '-c'
image: >-
nginx
name: output-artifact-gc
outputs:
artifacts:
- name: dontKeep1
path: /tmp/hello.txt
artifactGC:
strategy: OnWorkflowCompletion
s3:
...
- name: dontKeep2
path: /tmp/hello.txt
artifactGC:
strategy: OnWorkflowDeletion
s3:
...
```

### Logs from the workflow controller

```text
[controller] (starting) time="2025-04-11T10:25:16.365Z" level=info msg="Processing workflow" Phase= ResourceVersion=22122 namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.366Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:16.366Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:16.366Z" level=debug msg="Getting the template by name: output-artifact-gc" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:16.367Z" level=debug msg="Getting the template by name: output-artifact-gc" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:16.367Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:16.367Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:16.367Z" level=debug msg="Getting the template by name: output-artifact-gc" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:16.370Z" level=info msg="adding artifact GC finalizer" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.370Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.370Z" level=debug msg="Skipping artifact GC" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=info msg="Updated phase -> Running" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=debug msg="Evaluating node test-argo-artifact-gc2: template: *v1alpha1.WorkflowStep (output-artifact-gc), boundaryID: " namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=debug msg="Getting the template by name: output-artifact-gc" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"test-argo-artifact-gc2\", UID:\"3e11dfe5-13d1-4b73-b0b3-2c348e046e06\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"22122\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowRunning' Workflow Running"
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=debug msg="Initializing node test-argo-artifact-gc2: template: *v1alpha1.WorkflowStep (output-artifact-gc), boundaryID: " namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=info msg="Pod node test-argo-artifact-gc2 initialized Pending" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=debug msg="Executing node test-argo-artifact-gc2 with container template: output-artifact-gc\n" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=debug namespace=argo needLocation=true workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.371Z" level=warning msg="couldn't get boundaryTemplate through nodeName test-argo-artifact-gc2" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.372Z" level=debug msg="Creating Pod: test-argo-artifact-gc2 (test-argo-artifact-gc2)" namespace=argo workflow=test-argo-artifact-gc2
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca6792c3bd8] () WorkflowRunning:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca6792c3bd8] () WorkflowRunning: Workflow Running
[port-forward] (running) [pods/test-argo-artifact-gc2] (Pending) :
[port-forward] (running) [pods/test-argo-artifact-gc2] (Pending) :
[controller] (starting) time="2025-04-11T10:25:16.389Z" level=info msg="Created pod: test-argo-artifact-gc2 (test-argo-artifact-gc2)" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.389Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.389Z" level=info msg=reconcileAgentPod namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16Z" level=info msg="add pod event" pod=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16Z" level=info msg="update pod event" pod=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.389Z" level=debug msg="Log changes patch: {\"metadata\":{\"annotations\":{\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"finalizers\":[\"workflows.argoproj.io/artifact-gc\"],\"labels\":{\"default-label\":\"thisLabelIsFromWorkflowDefaults\",\"workflows.argoproj.io/completed\":\"false\",\"workflows.argoproj.io/phase\":\"Running\"}},...
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca67a1275ab] () Scheduled: Successfully assigned argo/test-argo-artifact-gc2 to k3d-k3s-default-server-0
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca67a1275ab] () Scheduled:
[controller] (starting) time="2025-04-11T10:25:16Z" level=info msg="update pod event" pod=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:16.396Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=22134 workflow=test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2] (Waiting) PodInitializing: container "wait" is waiting:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca69148280d] () Pulled: Container image "quay.io/argoproj/argoexec:latest" already present on machine
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca69148280d] () Pulled:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca691e2a176] () Created: Created container init
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca691e2a176] () Created:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca69402c70f] () Started:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca69402c70f] () Started: Started container init
[controller] (starting) time="2025-04-11T10:25:16Z" level=info msg="update pod event" pod=test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2] (Waiting) PodInitializing: container "wait" is waiting:
[port-forward] (running) [pods/test-argo-artifact-gc2] init: time="2025-04-11T10:25:16.848Z" level=info msg="Starting Workflow Executor" version=untagged
[port-forward] (running) [pods/test-argo-artifact-gc2] init: time="2025-04-11T10:25:16.850Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
[port-forward] (running) [pods/test-argo-artifact-gc2] init: time="2025-04-11T10:25:16.850Z" level=info msg="Executor initialized" deadline="2025-04-11 10:30:16 +0000 UTC" includeScriptOutput=false namespace=argo podName=test-argo-artifact-gc2 templateName=output-artifact-gc version="&Version{Version:untagged,BuildDate:2025-04-04T09:55:00Z,GitCommit:9b7c0c4c4ce1ecb8e1d27df95d14ccef09cffe0f,GitTag:untagged,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
[port-forward] (running) [pods/test-argo-artifact-gc2] init: time="2025-04-11T10:25:16.972Z" level=info msg="Start loading input artifacts..."
[port-forward] (running) [pods/test-argo-artifact-gc2] init: time="2025-04-11T10:25:16.972Z" level=info msg="Alloc=9704 TotalAlloc=15230 Sys=22869 NumGC=4 Goroutines=4"
[controller] (starting) time="2025-04-11T10:25:17.389Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=22134 namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=debug msg="Skipping artifact GC" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=debug msg="node changed" namespace=argo new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=test-argo-artifact-gc2 old.message= old.phase=Pending old.progress=0/1 workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=debug msg="Evaluating node test-argo-artifact-gc2: template: *v1alpha1.WorkflowStep (output-artifact-gc), boundaryID: " namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=debug msg="Getting the template by name: output-artifact-gc" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=debug msg="Executing node test-argo-artifact-gc2 of Pod is Pending" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=debug msg="Executing node test-argo-artifact-gc2 with container template: output-artifact-gc\n" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=debug msg="Skipped pod test-argo-artifact-gc2 (test-argo-artifact-gc2) creation: already exists" namespace=argo podPhase=Pending workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=info msg=reconcileAgentPod namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:17.390Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"}],\"nodes\":{\"test-argo-artifact-gc2\":{\"hostNodeName\":\"k3d-k3s-default-server-0\",\"message\":\"PodInitializing\"}}}}"
[controller] (starting) time="2025-04-11T10:25:17.398Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=22139 workflow=test-argo-artifact-gc2
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca6d0d2fadb] () Pulled: Container image "quay.io/argoproj/argoexec:latest" already present on machine
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca6d0d2fadb] () Pulled:
[controller] (starting) time="2025-04-11T10:25:17Z" level=info msg="update pod event" pod=test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2] (Waiting) PodInitializing: container "wait" is waiting:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca6d20981d0] () Created:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca6d20981d0] () Created: Created container wait
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca6d47b1ace] () Started:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca6d47b1ace] () Started: Started container wait
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca6d482e710] () Pulling:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca6d482e710] () Pulling: Pulling image "nginx"
[port-forward] (running) [pods/test-argo-artifact-gc2/init] tailing logs stopped
[controller] (starting) time="2025-04-11T10:25:18.862Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=22139 namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:18.863Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="task result:\n&WorkflowTaskResult{ObjectMeta:{test-argo-artifact-gc2 argo fe244593-f96c-4be9-9223-147d668533f1 22145 1 2025-04-11 10:25:17 +0000 UTC map[workflows.argoproj.io/report-outputs-completed:false workflows.argoproj.io/workflow:test-argo-artifact-gc2] map[] ...
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="task result name:\ntest-argo-artifact-gc2" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="Marking task result incomplete test-argo-artifact-gc2" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="Skipping artifact GC" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="node unchanged" namespace=argo nodeID=test-argo-artifact-gc2 workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="Evaluating node test-argo-artifact-gc2: template: *v1alpha1.WorkflowStep (output-artifact-gc), boundaryID: " namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="Getting the template by name: output-artifact-gc" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="Executing node test-argo-artifact-gc2 of Pod is Pending" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="Executing node test-argo-artifact-gc2 with container template: output-artifact-gc\n" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=debug msg="Skipped pod test-argo-artifact-gc2 (test-argo-artifact-gc2) creation: already exists" namespace=argo podPhase=Pending workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:18.864Z" level=info msg=reconcileAgentPod namespace=argo workflow=test-argo-artifact-gc2
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca71dde9c89] () Pulled: Successfully pulled image "nginx" in 1.230675126s (1.230726501s including waiting)
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca71dde9c89] () Pulled:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca71e8c7aa0] () Created: Created container main
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca71e8c7aa0] () Created:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca721030733] () Started: Started container main
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca721030733] () Started:
[controller] (starting) time="2025-04-11T10:25:19Z" level=info msg="update pod event" pod=test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2] (Ready) :
[port-forward] (running) [pods/test-argo-artifact-gc2] main: time="2025-04-11T10:25:19.225Z" level=info msg="capturing logs" argo=true
[port-forward] (running) [pods/test-argo-artifact-gc2] main: time="2025-04-11T10:25:19.225Z" level=debug msg="ignore signal child exited" argo=true
[port-forward] (running) [pods/test-argo-artifact-gc2] main: time="2025-04-11T10:25:19.226Z" level=debug msg="ignore signal child exited" argo=true
[port-forward] (running) [pods/test-argo-artifact-gc2] main: time="2025-04-11T10:25:20.227Z" level=info msg="sub-process exited" argo=true error=""
[port-forward] (running) [pods/test-argo-artifact-gc2] main: time="2025-04-11T10:25:20.227Z" level=info msg="/tmp/hello.txt -> /var/run/argo/outputs/artifacts/tmp/hello.txt.tgz" argo=true
[port-forward] (running) [pods/test-argo-artifact-gc2] main: time="2025-04-11T10:25:20.227Z" level=info msg="Taring /tmp/hello.txt"
[port-forward] (running) [pods/test-argo-artifact-gc2] main: time="2025-04-11T10:25:20.228Z" level=info msg="/tmp/hello.txt -> /var/run/argo/outputs/artifacts/tmp/hello.txt.tgz" argo=true
[port-forward] (running) [pods/test-argo-artifact-gc2] main: time="2025-04-11T10:25:20.228Z" level=info msg="Taring /tmp/hello.txt"
[controller] (starting) time="2025-04-11T10:25:20.884Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=22139 namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.885Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.885Z" level=debug msg="task result:\n&WorkflowTaskResult{ObjectMeta:{test-argo-artifact-gc2 argo fe244593-f96c-4be9-9223-147d668533f1 22145 1 2025-04-11 10:25:17 +0000 UTC map[workflows.argoproj.io/report-outputs-completed:false workflows.argoproj.io/workflow:test-argo-artifact-gc2] map[] ...
[controller] (starting) time="2025-04-11T10:25:20.885Z" level=debug msg="task result name:\ntest-argo-artifact-gc2" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.885Z" level=debug msg="Marking task result incomplete test-argo-artifact-gc2" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.885Z" level=debug msg="Skipping artifact GC" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.885Z" level=debug msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=test-argo-artifact-gc2 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.886Z" level=debug msg="Evaluating node test-argo-artifact-gc2: template: *v1alpha1.WorkflowStep (output-artifact-gc), boundaryID: " namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.886Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:20.886Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:20.886Z" level=debug msg="Getting the template by name: output-artifact-gc" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (starting) time="2025-04-11T10:25:20.886Z" level=debug msg="Executing node test-argo-artifact-gc2 of Pod is Running" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.886Z" level=debug msg="Executing node test-argo-artifact-gc2 with container template: output-artifact-gc\n" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.886Z" level=debug msg="Skipped pod test-argo-artifact-gc2 (test-argo-artifact-gc2) creation: already exists" namespace=argo podPhase=Running workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.886Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.886Z" level=info msg=reconcileAgentPod namespace=argo workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.886Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"True\",\"type\":\"PodRunning\"}],\"nodes\":{\"test-argo-artifact-gc2\":{\"message\":null,\"phase\":\"Running\"}},\"taskResultsCompletionStatus\":{\"test-argo-artifact-gc2\":false}}}"
[controller] (starting) time="2025-04-11T10:25:20Z" level=info msg="update pod event" pod=test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2] (Ready) :
[controller] (starting) time="2025-04-11T10:25:20.901Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=22151 workflow=test-argo-artifact-gc2
[controller] (starting) time="2025-04-11T10:25:20.902Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"test-argo-artifact-gc2\", UID:\"3e11dfe5-13d1-4b73-b0b3-2c348e046e06\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"22151\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node test-argo-artifact-gc2"
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca787363f62] () WorkflowNodeRunning:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca787363f62] () WorkflowNodeRunning: Running node test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:20.941Z" level=info msg="Main container completed" error=""
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:20.941Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:20.941Z" level=info msg="No output parameters"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:20.941Z" level=info msg="Saving output artifacts"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:20.941Z" level=info msg="Staging artifact: dontKeep1"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:20.941Z" level=info msg="Copying /tmp/hello.txt from container base image layer to /tmp/argo/outputs/artifacts/dontKeep1.tgz"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:20.941Z" level=info msg="/var/run/argo/outputs/artifacts/tmp/hello.txt.tgz -> /tmp/argo/outputs/artifacts/dontKeep1.tgz"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:20.942Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/dontKeep1.tgz, key: aaaaaaaaaa/dontKeepGC1"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:20.950Z" level=info msg="Creating minio client using static credentials" endpoint=***
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:20.950Z" level=info msg="Saving file to s3" bucket=*** endpoint=*** key=aaaaaaaaaa/dontKeepGC1 path=/tmp/argo/outputs/artifacts/dontKeep1.tgz
[port-forward] (running) [pods/test-argo-artifact-gc2/main] tailing logs stopped
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.245Z" level=info msg="Save artifact" artifactName=dontKeep1 duration=303.4355ms error="" key=aaaaaaaaaa/dontKeepGC1
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.245Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/dontKeep1.tgz
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.245Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/dontKeep1.tgz"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.245Z" level=info msg="Staging artifact: dontKeep2"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.245Z" level=info msg="Copying /tmp/hello.txt from container base image layer to /tmp/argo/outputs/artifacts/dontKeep2.tgz"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.245Z" level=info msg="/var/run/argo/outputs/artifacts/tmp/hello.txt.tgz -> /tmp/argo/outputs/artifacts/dontKeep2.tgz"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.245Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/dontKeep2.tgz, key: aaaaaaaaaa/dontKeepGC2"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.250Z" level=info msg="Creating minio client using static credentials" endpoint=***
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.250Z" level=info msg="Saving file to s3" bucket=*** endpoint=*** key=aaaaaaaaaa/dontKeepGC2 path=/tmp/argo/outputs/artifacts/dontKeep2.tgz
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.399Z" level=info msg="Save artifact" artifactName=dontKeep2 duration=153.473167ms error="" key=aaaaaaaaaa/dontKeepGC2
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.399Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/dontKeep2.tgz
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.399Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/dontKeep2.tgz"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.399Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: test-argo-artifact-gc2/test-argo-artifact-gc2/main.log"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.399Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.399Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=test-argo-artifact-gc2/test-argo-artifact-gc2/main.log path=/tmp/argo/outputs/logs/main.log
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.405Z" level=info msg="Save artifact" artifactName=main-logs duration=6.026208ms error="" key=test-argo-artifact-gc2/test-argo-artifact-gc2/main.log
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.405Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.405Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.408Z" level=debug msg="Create workflowtaskresults 409"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.411Z" level=debug msg="Patch workflowtaskresults 200"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.412Z" level=info msg="Alloc=9077 TotalAlloc=22232 Sys=23381 NumGC=6 Goroutines=14"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.417Z" level=debug msg="Patch workflowtaskresults 200"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.417Z" level=info msg="Deadline monitor stopped"
[port-forward] (running) [pods/test-argo-artifact-gc2] wait: time="2025-04-11T10:25:21.417Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
[controller] (running) time="2025-04-11T10:25:21.895Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=22151 namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.897Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.900Z" level=debug msg="task result:\n&WorkflowTaskResult{ObjectMeta:{test-argo-artifact-gc2 argo fe244593-f96c-4be9-9223-147d668533f1 22155 2 2025-04-11 10:25:17 +0000 UTC map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:test-argo-artifact-gc2] map[] ...
[controller] (running) time="2025-04-11T10:25:21.900Z" level=debug msg="task result name:\ntest-argo-artifact-gc2" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.900Z" level=debug msg="Marking task result complete test-argo-artifact-gc2" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.900Z" level=debug msg="task-result changed" namespace=argo nodeID=test-argo-artifact-gc2 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.900Z" level=debug msg="Skipping artifact GC" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21Z" level=info msg="queueing pod for cleanup" action=terminateContainers namespace=argo podName=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.900Z" level=debug msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=test-argo-artifact-gc2 old.message= old.phase=Running old.progress=0/1 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.900Z" level=debug msg="Evaluating node test-argo-artifact-gc2: template: *v1alpha1.WorkflowStep (output-artifact-gc), boundaryID: " namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.900Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (running) time="2025-04-11T10:25:21.900Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (running) time="2025-04-11T10:25:21.900Z" level=debug msg="Getting the template by name: output-artifact-gc" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (running) time="2025-04-11T10:25:21.900Z" level=debug msg="Executing node test-argo-artifact-gc2 of Pod is Running" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.901Z" level=debug msg="Executing node test-argo-artifact-gc2 with container template: output-artifact-gc\n" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.901Z" level=debug msg="Skipped pod test-argo-artifact-gc2 (test-argo-artifact-gc2) creation: already exists" namespace=argo podPhase=Running workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.901Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.901Z" level=info msg=reconcileAgentPod namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.901Z" level=debug msg="Log changes patch: {\"status\":{\"nodes\":{\"test-argo-artifact-gc2\":{\"outputs\":{\"artifacts\":[{\"artifactGC\":{\"strategy\":\"OnWorkflowCompletion\"},\"name\":\"dontKeep1\",\"path\":\"/tmp/hello.txt\",\"s3\":{\"accessKeySecret\":
[controller] (running) time="2025-04-11T10:25:21Z" level=info msg="cleaning up pod" action=terminateContainers key=argo/test-argo-artifact-gc2/terminateContainers namespace=argo podName=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.906Z" level=info msg="https://0.0.0.0:36167/api/v1/namespaces/argo/pods/test-argo-artifact-gc2/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
[controller] (running) time="2025-04-11T10:25:21Z" level=info msg="update pod event" pod=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.907Z" level=debug msg="exec container command" command="[/var/run/argo/argoexec kill 15 1]" container=wait error="" namespace=argo pod=test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2] (Ready) :
[controller] (running) time="2025-04-11T10:25:21.917Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=22157 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:21.921Z" level=info msg="signaled container" container=wait error="unable to upgrade connection: container not found (\"wait\")" namespace=argo pod=test-argo-artifact-gc2 stderr="" stdout=""
[controller] (running) time="2025-04-11T10:25:21Z" level=info msg="queueing pod for cleanup after" action=killContainers after=3s namespace=argo podName=test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2/wait] tailing logs stopped
[controller] (running) time="2025-04-11T10:25:22.908Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=22157 namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22.908Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="task result:\n&WorkflowTaskResult{ObjectMeta:{test-argo-artifact-gc2 argo fe244593-f96c-4be9-9223-147d668533f1 22155 2 2025-04-11 10:25:17 +0000 UTC map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:test-argo-artifact-gc2] map[] ...
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="task result name:\ntest-argo-artifact-gc2" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="Marking task result complete test-argo-artifact-gc2" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="Skipping artifact GC" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22Z" level=info msg="queueing pod for cleanup" action=terminateContainers namespace=argo podName=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="node unchanged" namespace=argo nodeID=test-argo-artifact-gc2 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="Evaluating node test-argo-artifact-gc2: template: *v1alpha1.WorkflowStep (output-artifact-gc), boundaryID: " namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="Getting the template by name: output-artifact-gc" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="Executing node test-argo-artifact-gc2 of Pod is Running" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="Executing node test-argo-artifact-gc2 with container template: output-artifact-gc\n" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22.909Z" level=debug msg="Skipped pod test-argo-artifact-gc2 (test-argo-artifact-gc2) creation: already exists" namespace=argo podPhase=Running workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22.909Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22.909Z" level=info msg=reconcileAgentPod namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22Z" level=info msg="cleaning up pod" action=terminateContainers key=argo/test-argo-artifact-gc2/terminateContainers namespace=argo podName=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:22Z" level=info msg="queueing pod for cleanup after" action=killContainers after=3s namespace=argo podName=test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2] (Ready) :
[controller] (running) time="2025-04-11T10:25:22Z" level=info msg="update pod event" pod=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23Z" level=info msg="update pod event" pod=test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2] (Ready) :
[controller] (running) time="2025-04-11T10:25:23.947Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=22157 namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=debug msg="task result:\n&WorkflowTaskResult{ObjectMeta:{test-argo-artifact-gc2 argo fe244593-f96c-4be9-9223-147d668533f1 22155 2 2025-04-11 10:25:17 +0000 UTC ...
[controller] (running) time="2025-04-11T10:25:23.948Z" level=debug msg="task result name:\ntest-argo-artifact-gc2" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=debug msg="Marking task result complete test-argo-artifact-gc2" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=debug msg="Skipping artifact GC" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=debug msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=test-argo-artifact-gc2 old.message= old.phase=Running old.progress=0/1 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=debug msg="Evaluating node test-argo-artifact-gc2: template: *v1alpha1.WorkflowStep (output-artifact-gc), boundaryID: " namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (running) time="2025-04-11T10:25:23.948Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (running) time="2025-04-11T10:25:23.948Z" level=debug msg="Getting the template by name: output-artifact-gc" base="*v1alpha1.Workflow (namespace=argo,name=test-argo-artifact-gc2)" tmpl="*v1alpha1.WorkflowStep (output-artifact-gc)"
[controller] (running) time="2025-04-11T10:25:23.948Z" level=debug msg="Node test-argo-artifact-gc2 already completed" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=info msg=reconcileAgentPod namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=debug msg="Task results completion status: map[test-argo-artifact-gc2:true]" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.948Z" level=info msg="Marking workflow completed" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.949Z" level=debug msg="Log changes patch: {\"metadata\":{\"labels\":{\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Succeeded\"}},\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"},{\"status\":\"True\",\"type\":\"Completed\"}],\"finishedAt\":\"2025-04-11T10:25:23Z\",\"nodes\":{\"test-argo-artifact-gc2\":{\"finishedAt\":\"2025-04-11T10:25:21Z\",\"phase\":\"Succeeded\",\"resourcesDuration\":{\"cpu\":0,\"memory\":3}}},\"phase\":\"Succeeded\"}}"
[controller] (running) time="2025-04-11T10:25:23.949Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"test-argo-artifact-gc2\", UID:\"3e11dfe5-13d1-4b73-b0b3-2c348e046e06\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"22157\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowSucceeded' Workflow completed"
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca83cd66057] () WorkflowSucceeded:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca83cd66057] () WorkflowSucceeded: Workflow completed
[controller] (running) time="2025-04-11T10:25:23.957Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=22162 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23.957Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"test-argo-artifact-gc2\", UID:\"3e11dfe5-13d1-4b73-b0b3-2c348e046e06\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"22162\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node test-argo-artifact-gc2"
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca83d581b70] () WorkflowNodeSucceeded:
[port-forward] (running) [events/test-argo-artifact-gc2.18353ca83d581b70] () WorkflowNodeSucceeded: Succeeded node test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=argo podName=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/test-argo-artifact-gc2/labelPodCompleted namespace=argo podName=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23Z" level=info msg="delete pod event" pod=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23Z" level=info msg="queuing poddelay1m56.022442737s" action=removeFinalizer namespace=argo podGC="{ nil }" podName=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:23Z" level=info msg="queueing pod for cleanup after" action=removeFinalizer after=1m56.022442737s namespace=argo podName=test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2] (Ready) :
[controller] (running) time="2025-04-11T10:25:24Z" level=info msg="cleaning up pod" action=killContainers key=argo/test-argo-artifact-gc2/killContainers namespace=argo podName=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:24.980Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=22162 namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:24.981Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:24.981Z" level=debug msg="processing Artifact GC Strategy OnWorkflowCompletion" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:24.981Z" level=debug msg="processing Artifact GC Strategy OnWorkflowCompletion" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:24.981Z" level=debug msg="list of artifacts pertaining to template output-artifact-gc to WorkflowArtifactGCTask \"test-argo-artifact-gc2-artgc-wfcomp-2166136261-0\": map[test-argo-artifact-gc2:{ArchiveLocation:&ArtifactLocation{ArchiveLogs:...
[controller] (running) time="2025-04-11T10:25:24.981Z" level=info msg="Creating Artifact GC Task test-argo-artifact-gc2-artgc-wfcomp-2166136261-0" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:24.990Z" level=info msg="creating pod to delete artifacts: test-argo-artifact-gc2-artgc-wfcomp-2166136261" namespace=argo strategy=OnWorkflowCompletion workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:24Z" level=info msg="add pod event" pod=test-argo-artifact-gc2-artgc-wfcomp-2166136261
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] (Pending) :
[controller] (running) time="2025-04-11T10:25:25Z" level=info msg="update pod event" pod=test-argo-artifact-gc2-artgc-wfcomp-2166136261
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] (Pending) :
[port-forward] (running) [events/test-argo-artifact-gc2-artgc-wfcomp-2166136261.18353ca87b7cfdb4] () Scheduled:
[port-forward] (running) [events/test-argo-artifact-gc2-artgc-wfcomp-2166136261.18353ca87b7cfdb4] () Scheduled: Successfully assigned argo/test-argo-artifact-gc2-artgc-wfcomp-2166136261 to k3d-k3s-default-server-0
[controller] (running) time="2025-04-11T10:25:25.005Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=22171 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:25Z" level=info msg="update pod event" pod=test-argo-artifact-gc2-artgc-wfcomp-2166136261
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] (Waiting) ContainerCreating: container "main" is waiting:
[port-forward] (running) [events/test-argo-artifact-gc2-artgc-wfcomp-2166136261.18353ca89d66f174] () Pulled:
[port-forward] (running) [events/test-argo-artifact-gc2-artgc-wfcomp-2166136261.18353ca89d66f174] () Pulled: Container image "quay.io/argoproj/argoexec:latest" already present on machine
[port-forward] (running) [events/test-argo-artifact-gc2-artgc-wfcomp-2166136261.18353ca89e210c08] () Created: Created container main
[port-forward] (running) [events/test-argo-artifact-gc2-artgc-wfcomp-2166136261.18353ca89e210c08] () Created:
[port-forward] (running) [events/test-argo-artifact-gc2-artgc-wfcomp-2166136261.18353ca8a5df2c30] () Started:
[port-forward] (running) [events/test-argo-artifact-gc2-artgc-wfcomp-2166136261.18353ca8a5df2c30] () Started: Started container main
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] (Ready) :
[controller] (running) time="2025-04-11T10:25:25Z" level=info msg="update pod event" pod=test-argo-artifact-gc2-artgc-wfcomp-2166136261
[controller] (running) time="2025-04-11T10:25:25.998Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=22171 namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:25.999Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=test-argo-artifact-gc2
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] main: time="2025-04-11T10:25:26.706Z" level=info msg="S3 Delete artifact: key: aaaaaaaaaa/dontKeepGC1"
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] main: time="2025-04-11T10:25:27.057Z" level=info msg="Creating minio client using static credentials" endpoint=***
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] main: time="2025-04-11T10:25:27.057Z" level=info msg="Deleting object from s3" bucket=*** endpoint=*** key=aaaaaaaaaa/dontKeepGC1
[controller] (running) time="2025-04-11T10:25:27Z" level=info msg="update pod event" pod=test-argo-artifact-gc2-artgc-wfcomp-2166136261
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] (Ready) :
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261/main] tailing logs stopped
[controller] (running) time="2025-04-11T10:25:28.967Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=22171 namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:28.968Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:29Z" level=info msg="update pod event" pod=test-argo-artifact-gc2-artgc-wfcomp-2166136261
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] (Ready) :
[controller] (running) time="2025-04-11T10:25:29Z" level=info msg="update pod event" pod=test-argo-artifact-gc2-artgc-wfcomp-2166136261
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] (Ready) :
[controller] (running) time="2025-04-11T10:25:30.060Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=22171 namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:30.061Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:30.061Z" level=info msg="reconciling artifact-gc pod" message= namespace=argo phase=Succeeded pod=test-argo-artifact-gc2-artgc-wfcomp-2166136261 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:30.061Z" level=info msg="processing completed Artifact GC Pod \"test-argo-artifact-gc2-artgc-wfcomp-2166136261\"" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:30.066Z" level=debug msg="processing WorkflowArtifactGCTask test-argo-artifact-gc2-artgc-wfcomp-2166136261-0" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:30.066Z" level=debug msg="deleting WorkflowArtifactGCTask: test-argo-artifact-gc2-artgc-wfcomp-2166136261-0" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:30.073Z" level=info msg="no remaining artifacts to GC, removing artifact GC finalizer (forceFinalizerRemoval=true)" namespace=argo workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:30.074Z" level=debug msg="Log changes patch: {\"metadata\":{\"finalizers\":null},\"status\":{\"artifactGCStatus\":{\"podsRecouped\":{\"test-argo-artifact-gc2-artgc-wfcomp-2166136261\":true}},\"nodes\":{\"test-argo-artifact-gc2\":{\"outputs\":{\"artifacts\":[{\"artifactGC\":{\"strategy\":\"OnWorkflowCompletion\"},\"deleted\":true,\"name\":\"dontKeep1\",\"path\":\"/tmp/hello.txt\",\"s3\":{...
[controller] (running) time="2025-04-11T10:25:30.083Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=22184 workflow=test-argo-artifact-gc2
[controller] (running) time="2025-04-11T10:25:30Z" level=info msg="update pod event" pod=test-argo-artifact-gc2-artgc-wfcomp-2166136261
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] (Deleting) :
[controller] (running) time="2025-04-11T10:25:30Z" level=info msg="delete pod event" pod=test-argo-artifact-gc2-artgc-wfcomp-2166136261
[port-forward] (running) [pods/test-argo-artifact-gc2-artgc-wfcomp-2166136261] (Deleting) :
[controller] (running) time="2025-04-11T10:25:31.086Z" level=debug msg="Won't process Workflow since it's completed" key=argo/test-argo-artifact-gc2
```

### Logs from in your workflow's wait container

```text
time="2025-04-11T12:16:34.825Z" level=info msg="Starting Workflow Executor" version=untagged
time="2025-04-11T12:16:34.827Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-04-11T12:16:34.827Z" level=info msg="Executor initialized" deadline="2025-04-11 12:21:33 +0000 UTC" includeScriptOutput=false namespace=argo podName=test-argo-artifact-gc2 templateName=output-artifact-gc version="&Version{Version:untagged,BuildDate:2025-04-04T09:55:00Z,GitCommit:9b7c0c4c4ce1ecb8e1d27df95d14ccef09cffe0f,GitTag:untagged,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-04-11T12:16:34.831Z" level=debug msg="Create workflowtaskresults 201"
time="2025-04-11T12:16:34.831Z" level=info msg="Starting deadline monitor"
time="2025-04-11T12:16:38.838Z" level=info msg="Main container completed" error=""
time="2025-04-11T12:16:38.838Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-04-11T12:16:38.838Z" level=info msg="No output parameters"
time="2025-04-11T12:16:38.838Z" level=info msg="Saving output artifacts"
time="2025-04-11T12:16:38.838Z" level=info msg="Staging artifact: dontKeep1"
time="2025-04-11T12:16:38.838Z" level=info msg="Copying /tmp/hello.txt from container base image layer to /tmp/argo/outputs/artifacts/dontKeep1.tgz"
time="2025-04-11T12:16:38.838Z" level=info msg="/var/run/argo/outputs/artifacts/tmp/hello.txt.tgz -> /tmp/argo/outputs/artifacts/dontKeep1.tgz"
time="2025-04-11T12:16:38.838Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/dontKeep1.tgz, key: aaaaaaaaaa/dontKeepGC1"
time="2025-04-11T12:16:38.846Z" level=info msg="Creating minio client using static credentials" endpoint=***
time="2025-04-11T12:16:38.846Z" level=info msg="Saving file to s3" bucket=*** endpoint=*** key=aaaaaaaaaa/dontKeepGC1 path=/tmp/argo/outputs/artifacts/dontKeep1.tgz
time="2025-04-11T12:16:38.915Z" level=info msg="Deadline monitor stopped"
time="2025-04-11T12:16:38.915Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2025-04-11T12:16:39.227Z" level=info msg="Save artifact" artifactName=dontKeep1 duration=388.613375ms error="" key=aaaaaaaaaa/dontKeepGC1
time="2025-04-11T12:16:39.227Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/dontKeep1.tgz
time="2025-04-11T12:16:39.227Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/dontKeep1.tgz"
time="2025-04-11T12:16:39.227Z" level=info msg="Staging artifact: dontKeep2"
time="2025-04-11T12:16:39.227Z" level=info msg="Copying /tmp/hello.txt from container base image layer to /tmp/argo/outputs/artifacts/dontKeep2.tgz"
time="2025-04-11T12:16:39.227Z" level=info msg="/var/run/argo/outputs/artifacts/tmp/hello.txt.tgz -> /tmp/argo/outputs/artifacts/dontKeep2.tgz"
time="2025-04-11T12:16:39.228Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/dontKeep2.tgz, key: aaaaaaaaaa/dontKeepGC2"
time="2025-04-11T12:16:39.235Z" level=info msg="Creating minio client using static credentials" endpoint=***
time="2025-04-11T12:16:39.236Z" level=info msg="Saving file to s3" bucket=*** endpoint=*** key=aaaaaaaaaa/dontKeepGC2 path=/tmp/argo/outputs/artifacts/dontKeep2.tgz
time="2025-04-11T12:16:39.403Z" level=info msg="Save artifact" artifactName=dontKeep2 duration=175.869333ms error="" key=aaaaaaaaaa/dontKeepGC2
time="2025-04-11T12:16:39.404Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/dontKeep2.tgz
time="2025-04-11T12:16:39.404Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/dontKeep2.tgz"
time="2025-04-11T12:16:39.405Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: test-argo-artifact-gc2/test-argo-artifact-gc2/main.log"
time="2025-04-11T12:16:39.405Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2025-04-11T12:16:39.405Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=test-argo-artifact-gc2/test-argo-artifact-gc2/main.log path=/tmp/argo/outputs/logs/main.log
time="2025-04-11T12:16:39.424Z" level=info msg="Save artifact" artifactName=main-logs duration=19.431625ms error="" key=test-argo-artifact-gc2/test-argo-artifact-gc2/main.log
time="2025-04-11T12:16:39.424Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2025-04-11T12:16:39.424Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2025-04-11T12:16:39.434Z" level=debug msg="Create workflowtaskresults 409"
time="2025-04-11T12:16:39.440Z" level=debug msg="Patch workflowtaskresults 200"
time="2025-04-11T12:16:39.441Z" level=info msg="Alloc=8357 TotalAlloc=22305 Sys=23893 NumGC=6 Goroutines=11"
time="2025-04-11T12:16:39.447Z" level=debug msg="Patch workflowtaskresults 200"
```

Start by reproducing the minimal workflow with two output artifacts, OnWorkflowCompletion and OnWorkflowDeletion strategies, and artifactGC.forceFinalizerRemoval enabled. Follow the workflow controller's artifact GC reconciliation and finalizer logs; done means the completion artifact is collected after completion and the deletion artifact remains until workflow deletion, then is cleaned up.

Written by the indexing model from the issue text.