argoproj / argoproj/argo-workflows

Artifact GC: a failed GC attempt is recorded as processed and never retried, leaving the finalizer permanently

Open
#16,894 0 comments 0 reactions 0 assignees View on GitHub
area/artifacts area/gc type/bug
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

### Pre-requisites

- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag and can confirm the issue still exists on `:latest` (the code path is unchanged on `main` at 38a9522aa8)
- [x] I have searched existing issues and could not find a match for this bug
- [x] I'd like to contribute the fix myself

### What happened? What did you expect to happen?

When a workflow completes with an artifact GC strategy configured, the controller adds the
`workflows.argoproj.io/artifact-gc` finalizer and then, on the next reconcile, runs
`processArtifactGCStrategy` to create the `WorkflowArtifactGCTask`s and the GC pod.

That function marks the strategy as processed in a `defer`, so it runs whether or not the
function returned an error:

```go
// workflow/controller/artifact_gc.go:127
func (woc *wfOperationCtx) processArtifactGCStrategy(ctx context.Context, strategy wfv1.ArtifactGCStrategy) error {
defer func() {
woc.wf.Status.ArtifactGCStatus.SetArtifactGCStrategyProcessed(strategy, true)
woc.updated = true
}()
```

If anything inside it fails (task create, pod create, template lookup), `operate` logs
`failed to GC artifacts` and returns, and the deferred `persistUpdates` writes
`strategiesProcessed.: true` to the workflow. From then on
`artifactGCStrategiesReady` (`artifact_gc.go:107`) considers the strategy done and never attempts
it again. Nothing ever marks the artifacts deleted, so `allArtifactsDeleted()` stays false and the
finalizer is never removed.

`forceFinalizerRemoval: true` does not help. It removes the finalizer only when
`AllArtifactGCPodsRecouped()` is true, and that returns false when `podsRecouped` is nil
(`pkg/apis/workflow/v1alpha1/workflow_types.go:1644`). A pod is only added to `podsRecouped`
when it is recouped (`artifact_gc.go:582`), not when it is created, so a workflow whose GC pod was
never created, or was created but never finished, has an empty map and the force flag can never fire.

The workflow ends up:

- phase Succeeded/Failed, `workflows.argoproj.io/completed=true`, archived
- `metadata.finalizers: [workflows.argoproj.io/artifact-gc]`
- `status.artifactGCStatus.strategiesProcessed.OnWorkflowCompletion: true`, no `podsRecouped`
- if deleted, stuck in Terminating with `OnWorkflowDeletion: true` as well

It stays like that forever. Because `reconciliationNeeded` is true while the finalizer is present,
the controller reconciles it every resync period doing nothing, and it keeps its parallelism slot
in the throttler. In the case that led me here, roughly 500 such workflows in one namespace had
consumed the entire `namespaceParallelism` budget, and every new workflow in that namespace was
being marked Pending with "Workflow processing has been postponed because too many workflows are
already running" indefinitely.

Any transient failure is enough to trigger this. Observed causes so far:

- API server pressure during a controller restart storm (429s / timeouts on the task or pod create)
- an admission webhook or `ValidatingAdmissionPolicy` rejecting the GC pod (#14355 describes this)
- a retry after an update conflict failing with `metadata.ownerReferences.uid: Invalid value: "": must not be empty` on the GC pod

Expected behaviour: a failed GC attempt should be retried on subsequent reconciles, ideally with
backoff. The strategy should only be recorded as processed once the tasks and pod actually exist.
If the attempts are exhausted, the workflow should carry an `ArtifactGCError` condition so
`forceFinalizerRemoval` (or an operator) can release it, rather than silently holding the finalizer.

Related symptoms already filed: #10840, #12316, #14355, #16599. Each describes one trigger for the
same end state.

### Version(s)

v4.0.1 observed in the field. Code path is identical on `main`.

### Paste a minimal workflow that reproduces the issue

Any workflow with an output artifact and `artifactGC.strategy: OnWorkflowCompletion`, run on a
cluster where GC pod creation fails once. The Kyverno policy in #14355 is a reliable way to make the
pod create fail; alternatively point the controller at an API server that returns 429 for the
first pod create.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: artgc-fail-once-
spec:
entrypoint: main
artifactGC:
strategy: OnWorkflowCompletion
templates:
- name: main
container:
image: alpine:3.20
command: [sh, -c, "echo hello > /tmp/out.txt"]
outputs:
artifacts:
- name: out
path: /tmp/out.txt
s3:
key: artgc-fail-once/out.txt
```

After the single failed attempt, remove the deny policy (or let the API server recover) and observe
that the controller never creates the GC pod, the finalizer remains, and `kubectl delete wf` hangs.

### Logs from the workflow controller

```
{"level":"INFO","msg":"Marking workflow completed", ...}
{"level":"INFO","msg":"adding artifact GC finalizer", ...}
{"level":"INFO","msg":"Creating Artifact GC Task", ...}
{"level":"INFO","msg":"creating pod to delete artifacts", ...}
{"level":"ERROR","msg":"failed to GC artifacts","error":"failed to create pod: Pod \"...-artgc-wfcomp-671565822\" is invalid: metadata.ownerReferences.uid: Invalid value: \"\": must not be empty", ...}
{"level":"INFO","msg":"Workflow update successful", ...}
# then every 20 minutes, forever:
{"level":"INFO","msg":"Processing workflow","Phase":"Succeeded", ...}
{"level":"INFO","msg":"Task-result reconciliation","numObjs":0, ...}
```

### Logs from in your workflow's wait container

Not applicable, the workflow's own pods completed normally.

Contributor guide

Open the contributing guide

Research direction

Start in workflow/controller/artifact_gc.go with processArtifactGCStrategy and artifactGCStrategiesReady, then inspect workflow_types.go around ArtifactGCStatus and podsRecouped. Trace how errors are persisted during reconciliation and how forceFinalizerRemoval uses that state. Done means a failed attempt can be retried or reported as an ArtifactGCError, without leaving the finalizer permanently.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.