grafana / grafana/plugin-ci-workflows

CD staleness check can reject newer code when an older release finishes later

Open
#1,037 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
10
Forks
6
Avg merge
19h 51m
Merged PRs (30d)
97

Description

We observed an overlapping tagged-release deployment and automatic main-branch deployment leave an environment on the older code.

Let A be the tagged release commit and B a subsequent commit on main. Both deployments target the same plugin and environment:

```text
Time →
t1 t2 t3 t4

Release A: starts ───────────────── updates
env to A
finishes

Main build B: starts ───────────────── checks → skipped


B started before A updated the config,
so B is classified as stale.

Source order: A → B (B contains newer code)
Result: environment stays on A.
```

Here, “updates env to A” means updating the environment's version configuration, not the exact time all running instances switch versions.

The staleness check compares the environment configuration file's latest commit timestamp with the deployment workflow's creation timestamp. In this sequence, it treats **“configuration changed after this workflow started”** as evidence that **“this deployment has been superseded.”**

But that configuration change came from the older release, not from a deployment containing B or a later commit.

There is a per-plugin/environment mutex, but preventing concurrent deployment steps does not establish source revision order. The failure above does not require clock skew.

How is deployment ordering intended to work when tagged-release and main-branch workflows overlap? In this case, the newer main build never reached the environment.

Related:
- https://github.com/grafana/plugin-ci-workflows/issues/207
- https://github.com/grafana/plugin-ci-workflows/issues/252

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the deployment workflow's staleness check and the per-plugin/environment mutex described in the issue, then compare their behavior for overlapping tagged-release and main-branch runs. Confirm the failure sequence with commits A and B; done means the newer main build is not classified as stale because an older release updated the environment configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, devops, release
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.