History-limit pruning deletes TaskRuns owned by a still-running PipelineRun, causing the pipeline task to re-run
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 6
- Forks
- 19
- Avg merge
- 12h 36m
- Merged PRs (30d)
- 27
Description
Expected Behavior
A TaskRun owned by a PipelineRun must never be pruned independently of its parent. Child TaskRuns are meant to be reclaimed through ownerReference cascade when the parent PipelineRun itself is deleted.
The codebase already states this intent in three places:
pkg/reconciler/taskrun/controller.go:83,92— the informer event filter usesisStandaloneTaskRunto drop any TaskRun carrying atekton.dev/pipelineRunlabel or a PipelineRun ownerReference.pkg/reconciler/taskrun/reconciler.go:45—ReconcileKindchecks it again, with the comment "will be handled by it is parent resource(PipelineRun)".pkg/reconciler/tektonpruner/controller.go:303— the periodic GC only processes TaskRuns where!HasPipelineRunOwnerReference(), and the function comment states "checks if the TaskRun ... is not owned by a PipelineRun before processing".
Count-based cleanup (history limit) should honour the same rule.
Actual Behavior
Those ownership filters only gate which TaskRun triggers the limiter. They do not constrain what gets deleted. The deletion candidate set is re-listed from the API server with no ownership filter at all.
pkg/config/history_limiter.go:274-342— for global- and namespace-level configuration (including the shipped default), theswitch identifiedByfalls through todefault:, which runshl.resourceFn.List(ctx, resource.GetNamespace(), "")— every TaskRun in the namespace.pkg/reconciler/taskrun/reconciler.go:124-136—TrFuncs.Listis a plainTaskRuns(namespace).List(...). It excludes neithertekton.dev/pipelineRunnor TaskRuns with a PipelineRun ownerReference.- The only subsequent filter is completion status (
history_limiter.go:349-355). A completed child TaskRun of a still-running PipelineRun passes it. - The list is sorted newest-first (
L362-371) and everything pasthistoryLimitis deleted (L374-421).
Consequence: the deleted child TaskRun is recreated and the pipeline task runs again. On the tektoncd/pipeline side (verified against v1.14.1):
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go:832-837—setTaskRunsAndResolvedTaskswallowsIsNotFound, sot.TaskRunsstays empty.- Same file,
L395-400—isScheduled()returnslen(t.TaskRuns) > 0, i.e.false.isSuccessful,isFailureandIsRunningshort-circuit onlen == 0in the same way. - The pipeline task is therefore treated as never started, re-enters the DAG execution queue, and is recreated by
createTaskRuns(pkg/reconciler/pipelinerun/pipelinerun.go:1111).
The task executes a second time and its previous results and status are lost. For non-idempotent tasks — publishing artifacts, provisioning VMs, pushing tags — this is considerably worse than wasted compute.
This affects the out-of-the-box configuration. The shipped config/600-tekton-pruner-default-spec.yaml is enforcedConfigLevel: global with historyLimit: 100, which is exactly the path that lists the whole namespace without an ownership filter.
One precondition is worth stating clearly: this path only fires when a standalone TaskRun completes in the namespace. But a single such trigger sweeps all completed TaskRuns in that namespace beyond historyLimit, children of running PipelineRuns included.
Steps to Reproduce the Problem
-
Install the pruner and shrink the limit so the effect is quick to observe:
apiVersion: v1 kind: ConfigMap metadata: name: tekton-pruner-default-spec namespace: tekton-pipelines data: global-config: | enforcedConfigLevel: global successfulHistoryLimit: 2 -
In namespace
demo, start a long PipelineRun whose first few tasks finish quickly and whose last task sleeps for 10 minutes. Wait until at least 3 of its child TaskRuns have completed successfully while the PipelineRun is still running. -
In the same namespace, create and complete one standalone TaskRun (a bare TaskRun not owned by any PipelineRun; a single
truestep is enough). This is the only thing needed to trigger the TaskRun history limiter. -
Observe that the pruner deletes every successful TaskRun in
demobeyond the newest 2 — including the completed children of the still-running PipelineRun. -
The PipelineRun controller then recreates the deleted TaskRuns and the corresponding pipeline tasks execute again.
Note that step 4 is invisible at the default log level, because deletions are only logged at DEBUG (see #283). To observe it, either raise the log level or watch the tekton_pruner_controller_resources_deleted_total{resource_type="taskrun",operation="history"} metric.
Additional Info
Observed on a production build cluster (16-day window, pruner pod with no restarts, so the counters cover the full window):
tekton_pruner_controller_resources_deleted_total{resource_type="taskrun",operation="history"}totals 884 across 5 namespaces, with 377 in the busiest one.- The age-at-deletion histogram is telling. In one namespace, of 237 history-path deletions, 162 (68%) happened less than 30 minutes after the TaskRun completed, 39 under 10 minutes, and 6 under 5 minutes. Build pipelines on that cluster routinely run longer than half an hour.
- Simulating
doResourceCleanupagainst live cluster state: in a namespace configured withsuccessfulHistoryLimit: 10, 12 TaskRuns were in the deletion set and 8 of them belonged to 2 PipelineRuns that were still running. One more standalone TaskRun completing in that namespace would have deleted all 8.
Still present on main (05bebae): the only occurrence of PipelineRun in pkg/config/history_limiter.go is the metrics resource type at L383; TrFuncs.List is unchanged; the only ownership filters in the repository remain the two trigger-side ones.
Related issues
- #283 (Log deletions at INFO level instead of DEBUG) explains why this has stayed invisible: a deleted TaskRun leaves no log line, and users only see a pipeline task inexplicably running twice, which is easily mistaken for a transient retry.
- #352 / #353 (garbage collection not serialized) is a different defect, but it widens the window: concurrent sweeps multiply the chance of hitting a running PipelineRun's children. That issue characterises concurrent sweeps as "rather than data corruption", which suggests it was not known at the time that a single count-based sweep can already delete children of a running PipelineRun.
Suggested fix
Exclude PipelineRun-owned TaskRuns from the candidate set in doResourceCleanup, reusing the existing isStandaloneTaskRun criteria:
- server-side and cheap: append the negative label selector
!tekton.dev/pipelineRunto TaskRun list calls; - client-side as a safety net: re-check the PipelineRun ownerReference on the listed results, covering objects where the label is absent.
No change is needed on the PipelineRun side — child TaskRuns are already reclaimed by ownerReference cascade when the parent is deleted.
- Kubernetes version:
Server Version: v1.34.5
- Tekton Pipeline version:
v1.6.0
- Tekton Pruner version:
v0.3.4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pkg/config/history_limiter.go at doResourceCleanup and compare its candidate listing with the ownership checks in pkg/reconciler/taskrun/controller.go and reconciler.go, including TrFuncs.List. Verify the history-limiter path against the reproduction scenario. Done means completed TaskRuns owned by a PipelineRun are excluded from count-based deletion while standalone TaskRuns remain eligible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100