ArtifactGC prevents Workflow deletion

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

### What happened

I applied an `artifactGC` strategy (`OnWorkflowDeletion`) to my WorkflowTemplate. Artifacts were deleted, but the Workflow generated from the template cannot be deleted.

### What I expected to happen

Artifacts would be deleted and the workflow would be able to be deleted

### Additional Notes

Originally discussed with @juliev0 in #10192, but was recommended to create a new issue [here](https://github.com/argoproj/argo-workflows/issues/10192#issuecomment-1492335617) for the following reasons:
- Issue is not resolved by setting `serviceAccountName` in `artifactGC`.
- `argo delete --force` is not working.

```
$k get wf | grep artifact-gc
artifact-gc-test-bz9fk Succeeded 8m8s

$ k get pods | grep artifact-gc
artifact-gc-test-bz9fk-artgc-wfdel-1494256823 0/1 Completed 0 8m29s

$ k logs artifact-gc-test-bz9fk-artgc-wfdel-1494256823
time="2023-04-06T14:43:34.236Z" level=info msg="S3 Delete artifact: key: output.json"
time="2023-04-06T14:43:34.236Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2023-04-06T14:43:34.446Z" level=info msg="Deleting object from s3" bucket=argo-workflows-artifacts endpoint=s3.us-west-1.amazonaws.com key=output.json
time="2023-04-06T14:43:34.492Z" level=info msg="S3 Delete artifact: key: artifact-gc-test-bz9fk/artifact-gc-test-bz9fk/main.log"
time="2023-04-06T14:43:34.492Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2023-04-06T14:43:34.521Z" level=info msg="Deleting object from s3" bucket=argo-workflows-artifacts endpoint=s3.us-west-1.amazonaws.com key=artifact-gc-test-bz9fk/artifact-gc-test-bz9fk/main.log

$ k describe wf artifact-gc-test-bz9fk
Name: artifact-gc-test-bz9fk
Namespace: argo-workflows
Labels: submit-from-ui=true
workflows.argoproj.io/completed=true
workflows.argoproj.io/controller-instanceid=argo-workflows
workflows.argoproj.io/creator=system-serviceaccount-argo-workflows-argo-workflows-server
workflows.argoproj.io/phase=Succeeded
workflows.argoproj.io/workflow-archiving-status=Archived
workflows.argoproj.io/workflow-template=artifact-gc-test
Annotations: workflows.argoproj.io/pod-name-format: v2
API Version: argoproj.io/v1alpha1
Kind: Workflow
Metadata:
Creation Timestamp: 2023-04-06T14:43:11Z
Deletion Grace Period Seconds: 0
Deletion Timestamp: 2023-04-06T14:43:27Z
Finalizers:
workflows.argoproj.io/artifact-gc
Generate Name: artifact-gc-test-
Generation: 5
Managed Fields:
API Version: argoproj.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:generateName:
f:labels:
.:
f:submit-from-ui:
f:workflows.argoproj.io/controller-instanceid:
f:workflows.argoproj.io/creator:
f:workflows.argoproj.io/workflow-template:
f:spec:
Manager: argo
Operation: Update
Time: 2023-04-06T14:43:11Z
API Version: argoproj.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:workflows.argoproj.io/pod-name-format:
f:finalizers:
.:
v:"workflows.argoproj.io/artifact-gc":
f:labels:
f:workflows.argoproj.io/completed:
f:workflows.argoproj.io/phase:
f:workflows.argoproj.io/workflow-archiving-status:
f:status:
Manager: workflow-controller
Operation: Update
Time: 2023-04-06T14:43:21Z
Resource Version: 895872955
UID: 2c80ccfa-53ef-4a0e-a47d-7b2a603f73f8
Spec:
Arguments:
Workflow Template Ref:
Name: artifact-gc-test
Status:
Artifact GC Status:
Strategies Processed:
On Workflow Completion: true
On Workflow Deletion: true
Artifact Repository Ref:
Artifact Repository:
Archive Logs: true
s3:
Bucket: argo-workflows-artifacts
Endpoint: s3.us-west-1.amazonaws.com
Insecure: false
Region: us-west-1
Use SDK Creds: true
Default: true
Conditions:
Status: False
Type: PodRunning
Status: True
Type: Completed
Finished At: 2023-04-06T14:43:21Z
Nodes:
artifact-gc-test-bz9fk:
Display Name: artifact-gc-test-bz9fk
Finished At: 2023-04-06T14:43:14Z
Host Node Name: XXX.compute.internal
Id: artifact-gc-test-bz9fk
Name: artifact-gc-test-bz9fk
Outputs:
Artifacts:
Name: file
Path: /tmp/output.json
s3:
Key: output.json
Name: main-logs
s3:
Key: artifact-gc-test-bz9fk/artifact-gc-test-bz9fk/main.log
Exit Code: 0
Phase: Succeeded
Progress: 1/1
Resources Duration:
Cpu: 4
Memory: 4
Started At: 2023-04-06T14:43:11Z
Template Name: generate-artifact
Template Scope: local/
Type: Pod
Phase: Succeeded
Progress: 1/1
Resources Duration:
Cpu: 4
Memory: 4
Started At: 2023-04-06T14:43:11Z
Stored Templates:
namespaced/artifact-gc-test/generate-artifact:
Container:
Args:
echo "Generating JSON file..."
echo '{"test":"json"}' > /tmp/output.json

Command:
sh
-c
Image: argoproj/argosay:v2
Name:
Resources:
Inputs:
Metadata:
Name: generate-artifact
Outputs:
Artifacts:
Name: file
Path: /tmp/output.json
s3:
Key: output.json
Stored Workflow Template Spec:
Arguments:
Artifact GC:
Service Account Name: argo-workflows-workflow
Strategy: OnWorkflowDeletion
Entrypoint: generate-artifact
Pod GC:
Strategy: OnWorkflowSuccess
Service Account Name: argo-workflows-workflow
Templates:
Container:
Args:
echo "Generating JSON file..."
echo '{"test":"json"}' > /tmp/output.json

Command:
sh
-c
Image: argoproj/argosay:v2
Name:
Resources:
Inputs:
Metadata:
Name: generate-artifact
Outputs:
Artifacts:
Name: file
Path: /tmp/output.json
s3:
Key: output.json
Ttl Strategy:
Seconds After Failure: 10800
Seconds After Success: 5
Workflow Template Ref:
Name: artifact-gc-test
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WorkflowRunning 13m workflow-controller Workflow Running
Normal WorkflowSucceeded 13m workflow-controller Workflow completed
Normal WorkflowNodeRunning 13m workflow-controller Running node artifact-gc-test-bz9fk
Normal WorkflowNodeSucceeded 13m workflow-controller Succeeded node artifact-gc-test-bz9fk
```

### Version

v3.4.5

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
metadata:
name: artifact-gc-test
namespace: argo-workflows
spec:
templates:
- name: generate-artifact
outputs:
artifacts:
- name: file
path: /tmp/output.json
s3:
key: output.json
container:
image: argoproj/argosay:v2
command:
- sh
- '-c'
args:
- |
echo "Generating JSON file..."
echo '{"test":"json"}' > /tmp/output.json
entrypoint: generate-artifact
serviceAccountName: argo-workflows-workflow
artifactGC:
strategy: OnWorkflowDeletion
serviceAccountName: argo-workflows-workflow
```
```

### Logs from the workflow controller

```text
$ k logs deploy/argo-workflows-workflow-controller | grep artifact-gc-test-bz9fk
time="2023-04-06T14:43:11.022Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:11.036Z" level=info msg="adding artifact GC finalizer" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:11.036Z" level=info msg="Updated phase -> Running" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:11.067Z" level=info msg="Pod node artifact-gc-test-bz9fk initialized Pending" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:11.115Z" level=info msg="Created pod: artifact-gc-test-bz9fk (artifact-gc-test-bz9fk)" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:11.115Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:11.115Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:11.124Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Running resourceVersion=895872537 workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.116Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.116Z" level=info msg="Task-result reconciliation" namespace=argo-workflows numObjs=1 workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.116Z" level=info msg="task-result changed" namespace=argo-workflows nodeID=artifact-gc-test-bz9fk workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.116Z" level=info msg="node changed" namespace=argo-workflows new.message= new.phase=Succeeded new.progress=0/1 nodeID=artifact-gc-test-bz9fk old.message= old.phase=Pending old.progress=0/1 workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.117Z" level=info msg="TaskSet Reconciliation" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.117Z" level=info msg=reconcileAgentPod namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.117Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.117Z" level=info msg="Marking workflow completed" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.117Z" level=info msg="Marking workflow as pending archiving" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.117Z" level=info msg="Checking daemoned children of " namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.122Z" level=info msg="cleaning up pod" action=deletePod key=argo-workflows/artifact-gc-test-bz9fk-1340600742-agent/deletePod
time="2023-04-06T14:43:21.128Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Succeeded resourceVersion=895872755 workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.140Z" level=info msg="archiving workflow" namespace=argo-workflows uid=2c80ccfa-53ef-4a0e-a47d-7b2a603f73f8 workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:21.176Z" level=info msg="Queueing Succeeded workflow argo-workflows/artifact-gc-test-bz9fk for delete in 5s due to TTL"
time="2023-04-06T14:43:26.140Z" level=info msg="cleaning up pod" action=deletePod key=argo-workflows/artifact-gc-test-bz9fk/deletePod
time="2023-04-06T14:43:27.000Z" level=info msg="Deleting garbage collected workflow 'argo-workflows/artifact-gc-test-bz9fk'"
time="2023-04-06T14:43:27.011Z" level=info msg="Successfully deleted 'argo-workflows/artifact-gc-test-bz9fk'"
time="2023-04-06T14:43:31.176Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:31.177Z" level=info msg="Creating Artifact GC Task artifact-gc-test-bz9fk-artgc-wfdel-1494256823-0" namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:31.188Z" level=info msg="creating pod to delete artifacts: artifact-gc-test-bz9fk-artgc-wfdel-1494256823" namespace=argo-workflows strategy=OnWorkflowDeletion workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:31.207Z" level=info msg="Checking daemoned children of " namespace=argo-workflows workflow=artifact-gc-test-bz9fk
time="2023-04-06T14:43:31.219Z" level=info msg="Workflow update successful" namespace=argo-workflows phase=Succeeded resourceVersion=895872955 workflow=artifact-gc-test-bz9fk
```
```

### Logs from in your workflow's wait container

```text
$ k logs -c wait -l workflows.argoproj.io/workflow=artifact-gc-test-bz9fk,workflow.argoproj.io/phase!=Succeeded
error: container wait is not valid for pod artifact-gc-test-bz9fk-artgc-wfdel-1494256823
```
```