argoproj / argoproj/argo-workflows

ArtifactGC failures due to denying pod webhooks are not properly reported

Open
#14,355 0 comments 0 reactions 0 assignees View on GitHub
area/artifacts area/gc
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

While running a workflow with artifactGC configured, I realized that argo does not properly identify that GC fails, when the artifactGC pod is denied by a webhook. For reproducing the issue, i used this kyverno policy that just denies every artifact gc pod:
```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: deny-argo-artifact-gc-pods
spec:
validationFailureAction: Enforce
rules:
- name: deny-argo-artifact-gc-pods
match:
any:
- resources:
kinds:
- Pod
selector:
matchLabels:
workflows.argoproj.io/component: artifact-gc
validate:
deny:
conditions:
any:
- key: "{{ request.object.metadata.labels.\"workflows.argoproj.io/component\" || '' }}"
operator: Equals
value: artifact-gc
message: "argo artifact-gc pod denied"
```
When now running a worklfow with artifactGC configured, the artifact gc pod is denied, which is also visible in the logs. But the the workflow status does not get the the artifactGC failed condition, its just:
```yaml
status:
artifactGCStatus:
strategiesProcessed:
OnWorkflowCompletion: true
artifactRepositoryRef:
artifactRepository:
archiveLogs: true
s3:
...
configMap: artifact-repositories
key: default-v1
namespace: argo
conditions:
- status: "False"
type: PodRunning
- status: "True"
type: Completed
finishedAt: "2025-04-04T12:04:11Z"
nodes:
test-argo-artifact-gc:
displayName: test-argo-artifact-gc
finishedAt: "2025-04-04T12:04:09Z"
hostNodeName: k3d-k3s-default-server-0
id: test-argo-artifact-gc
name: test-argo-artifact-gc
outputs:
artifacts:
- name: keep
path: /tmp/keep.txt
s3:
...
- name: main-logs
s3:
key: test-argo-artifact-gc/test-argo-artifact-gc/main.log
exitCode: "0"
phase: Succeeded
progress: 1/1
resourcesDuration:
cpu: 0
memory: 3
startedAt: "2025-04-04T12:04:04Z"
templateName: output-artifact-gc
templateScope: local/test-argo-artifact-gc
type: Pod
phase: Succeeded
progress: 1/1
resourcesDuration:
cpu: 0
memory: 3
startedAt: "2025-04-04T12:04:04Z"
taskResultsCompletionStatus:
test-argo-artifact-gc: true
```

Furthermore, even when running the workflow with `forceFinalizerRemoval: true`, the `workflows.argoproj.io/artifact-gc` finalizer does not get removed, even though artifactGC failed.

### Version(s)

v3.6.5 (latest main)

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: test-artifact-gc
namespace: argo
spec:
artifactGC:
forceFinalizerRemoval: true
strategy: OnWorkflowCompletion
entrypoint: output-artifact-gc
templates:
- container:
args:
- echo "keep" > /tmp/keep.txt
command:
- sh
- '-c'
image: >-
nginx
name: output-artifact-gc
outputs:
artifacts:
- name: keep
path: /tmp/keep.txt
s3:
...
```

### Logs from the workflow controller

```text
[port-forward] (running) [events/test-argo-artifact-gc.18331bfc7ef3534d] () WorkflowSucceeded: Workflow completed
[controller] (running) time="2025-04-04T12:04:12.005Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=13478 workflow=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:12.006Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"test-argo-artifact-gc\", UID:\"0328cf80-1406-4832-b31f-0e42ad9c1da1\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"13478\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node test-argo-artifact-gc"
[port-forward] (running) [events/test-argo-artifact-gc.18331bfc7f6bb45b] () WorkflowNodeSucceeded:
[port-forward] (running) [events/test-argo-artifact-gc.18331bfc7f6bb45b] () WorkflowNodeSucceeded: Succeeded node test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:12Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=argo podName=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:12Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/test-argo-artifact-gc/labelPodCompleted namespace=argo podName=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:12Z" level=info msg="delete pod event" pod=test-argo-artifact-gc
[port-forward] (running) [pods/test-argo-artifact-gc] (Ready) :
[controller] (running) time="2025-04-04T12:04:12Z" level=info msg="queuing poddelay1m55.925913164s" action=removeFinalizer namespace=argo podGC="{ nil }" podName=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:12Z" level=info msg="queueing pod for cleanup after" action=removeFinalizer after=1m55.925913164s namespace=argo podName=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:12Z" level=info msg="cleaning up pod" action=killContainers key=argo/test-argo-artifact-gc/killContainers namespace=argo podName=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:13.076Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=13478 namespace=argo workflow=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:13.079Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:13.079Z" level=debug msg="processing Artifact GC Strategy OnWorkflowCompletion" namespace=argo workflow=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:13.079Z" level=debug msg="processing Artifact GC Strategy OnWorkflowCompletion" namespace=argo workflow=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:13.080Z" level=debug msg="list of artifacts pertaining to template output-artifact-gc to WorkflowArtifactGCTask \"test-argo-artifact-gc-artgc-wfcomp-2166136261-0\": ***
[controller] (running) time="2025-04-04T12:04:13.080Z" level=info msg="Creating Artifact GC Task test-argo-artifact-gc-artgc-wfcomp-2166136261-0" namespace=argo workflow=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:13.099Z" level=info msg="creating pod to delete artifacts: test-argo-artifact-gc-artgc-wfcomp-2166136261" namespace=argo strategy=OnWorkflowCompletion workflow=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:13.117Z" level=error msg="failed to GC artifacts" error="failed to create pod: admission webhook \"validate.kyverno.svc-fail\" denied the request: \n\nresource Pod/argo/test-argo-artifact-gc-artgc-wfcomp-2166136261 was blocked due to the following policies \n\ndeny-argo-artifact-gc-pods:\n deny-argo-artifact-gc-pods: argo artifact-gc pod denied\n" namespace=argo workflow=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:13.130Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=13488 workflow=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:18.138Z" level=info msg="Processing workflow" Phase=Succeeded ResourceVersion=13488 namespace=argo workflow=test-argo-artifact-gc
[controller] (running) time="2025-04-04T12:04:18.141Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=test-argo-artifact-gc
```

### Logs from in your workflow's wait container

```text
time="2025-04-04T12:04:05.966Z" level=info msg="Starting Workflow Executor" version=untagged
time="2025-04-04T12:04:05.968Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-04-04T12:04:05.968Z" level=info msg="Executor initialized" deadline="2025-04-04 12:09:04 +0000 UTC" includeScriptOutput=false namespace=argo podName=test-argo-artifact-gc templateName=output-artifact-gc version="&Version{Version:untagged,BuildDate:2025-04-04T09:55:00Z,GitCommit:9b7c0c4c4ce1ecb8e1d27df95d14ccef09cffe0f,GitTag:untagged,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-04-04T12:04:05.974Z" level=debug msg="Create workflowtaskresults 201"
time="2025-04-04T12:04:05.975Z" level=info msg="Starting deadline monitor"
time="2025-04-04T12:04:08.978Z" level=info msg="Main container completed" error=""
time="2025-04-04T12:04:08.978Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-04-04T12:04:08.978Z" level=info msg="No output parameters"
time="2025-04-04T12:04:08.978Z" level=info msg="Saving output artifacts"
time="2025-04-04T12:04:08.978Z" level=info msg="Staging artifact: keep"
time="2025-04-04T12:04:08.978Z" level=info msg="Copying /tmp/keep.txt from container base image layer to /tmp/argo/outputs/artifacts/keep.tgz"
time="2025-04-04T12:04:08.978Z" level=info msg="/var/run/argo/outputs/artifacts/tmp/keep.txt.tgz -> /tmp/argo/outputs/artifacts/keep.tgz"
time="2025-04-04T12:04:08.978Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/keep.tgz, key: aaaaaaaaaa/keepGC2"
time="2025-04-04T12:04:08.992Z" level=info msg="Creating minio client using static credentials" endpoint=***
time="2025-04-04T12:04:08.992Z" level=info msg="Saving file to s3" bucket=*** endpoint=*** key=aaaaaaaaaa/keepGC2 path=/tmp/argo/outputs/artifacts/keep.tgz
time="2025-04-04T12:04:09.212Z" level=info msg="Save artifact" artifactName=keep duration=234.247708ms error="" key=aaaaaaaaaa/keepGC2
time="2025-04-04T12:04:09.213Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/artifacts/keep.tgz
time="2025-04-04T12:04:09.213Z" level=info msg="Successfully saved file: /tmp/argo/outputs/artifacts/keep.tgz"
time="2025-04-04T12:04:09.213Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: test-argo-artifact-gc/test-argo-artifact-gc/main.log"
time="2025-04-04T12:04:09.213Z" level=info msg="Creating minio client using static credentials" endpoint="minio:9000"
time="2025-04-04T12:04:09.213Z" level=info msg="Saving file to s3" bucket=my-bucket endpoint="minio:9000" key=test-argo-artifact-gc/test-argo-artifact-gc/main.log path=/tmp/argo/outputs/logs/main.log
time="2025-04-04T12:04:09.234Z" level=info msg="Save artifact" artifactName=main-logs duration=21.055375ms error="" key=test-argo-artifact-gc/test-argo-artifact-gc/main.log
time="2025-04-04T12:04:09.234Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2025-04-04T12:04:09.234Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2025-04-04T12:04:09.244Z" level=debug msg="Create workflowtaskresults 409"
time="2025-04-04T12:04:09.250Z" level=debug msg="Patch workflowtaskresults 200"
time="2025-04-04T12:04:09.251Z" level=info msg="Alloc=8435 TotalAlloc=20760 Sys=23125 NumGC=5 Goroutines=12"
time="2025-04-04T12:04:09.256Z" level=debug msg="Patch workflowtaskresults 200"
time="2025-04-04T12:04:09.257Z" level=info msg="Deadline monitor stopped"
time="2025-04-04T12:04:09.257Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
```

Contributor guide

Open the contributing guide

Research direction

Start with the workflow controller's Artifact GC processing path and reproduce the failure using the Kyverno denying-pod policy and the minimal workflow shown here. Done means a denied Artifact GC pod marks the workflow's artifactGCStatus as failed and removes the workflows.argoproj.io/artifact-gc finalizer when forceFinalizerRemoval is enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.