tektoncd / tektoncd/pipelines-as-code

queue rebuild counts finished PipelineRuns as running after a watcher restart

Open
#2,949 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
214
Forks
144
Avg merge
2d 11h
Merged PRs (30d)
27

Description

📝 Description

When the watcher starts, InitQueues rebuilds its in-memory picture of every
repository's queue from the PipelineRuns in the cluster. To decide whether a
run is occupying a slot it reads one thing: the Pipelines-as-Code state
annotation (pkg/queue/queue_manager.go:240). It never asks Tekton whether
the run already finished.

A PipelineRun that completed, but whose annotation never advanced from
started to completed, is rebuilt as a running run. It occupies a slot, and
since only a running run finishing releases a slot, that slot is gone until
the next restart, which rebuilds the same wrong picture again.

The annotation lags behind reality whenever the watcher fails or dies between
a run finishing and the final annotation write. The provider-detection issue
(see companion issue) is one concrete way to get there.

A unit test confirms it: a PipelineRun with a Succeeded condition and a stale
started annotation comes back from InitQueues inside the running queue.

Found during a post-merge review of #2890.

🛠️ Suggested fix

During the rebuild, skip any PipelineRun that Tekton reports as done or
cancelled, whatever its annotation says.

🧪 Testing Strategy

  • Unit tests: a completed run with a stale started annotation must not occupy a slot after InitQueues

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at InitQueues and the annotation handling around pkg/queue/queue_manager.go:240, then inspect how PipelineRun completion and cancellation are represented by Tekton. The work is done when completed or cancelled runs with stale annotations are absent from the running queue, and the existing unit test continues to pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.