NVIDIA / NVIDIA/nvcf

chart-version-bump and stack-pin-bump silently drop queued runs on simultaneous releases

Open
#1,762 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

What happened

When several service or chart releases land within the same window (for
example a combined-PR merge that tags multiple services at once), the
`chart-version-bump.yml` and `stack-pin-bump.yml` workflows can silently
drop some of the bumps they were supposed to apply.

Both workflows share a single, repo-wide, un-scoped concurrency group
(`chart-version-bump` / `stack-pin-bump`). GitHub Actions keeps at most one
queued run per concurrency group: when a new run is queued while another is
already queued (not yet started), the older queued run is cancelled before
it ever runs, with no error surfaced anywhere.

Real occurrence

On 2026-09-04, six services released within about two minutes as part of a
combined merge. Two `chart-version-bump.yml` runs were cancelled while
queued:

  • `src/control-plane-services/cloud-functions/v1.17.0`
  • `src/control-plane-services/encrypted-secret-store/v0.5.0`

Neither bump landed. `encrypted-secret-store`'s `Chart.yaml` `appVersion`
stayed stale until a manual `workflow_dispatch` re-run recovered it six days
later. Nothing in the pipeline surfaced the gap; both runs showed as
"cancelled" with zero jobs, easy to miss in the Actions history.

Confirmation

The cancelled runs state the cause themselves:

Canceling since a higher priority waiting request for chart-version-bump exists

GitHub names the concurrency group in the message, so this is the group evicting
a queued run rather than anything in the workflow body failing. Both runs record
zero jobs, consistent with cancellation before any job started.

Fix

Scope each workflow's concurrency group per release tag instead of
repo-wide, so simultaneous releases for different services or charts no
longer cancel each other's queued runs. Add a retry loop around the shared
bump branch's push, since per-tag groups mean truly concurrent runs can now
race on that branch; force-with-lease turns a race into a rejected push
that the retry recovers from, rather than a silent drop.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting .github/workflows/chart-version-bump.yml and .github/workflows/stack-pin-bump.yml, then trace the shared bump branch push logic. Scope each workflow's concurrency group by release tag and add retry handling for rejected force-with-lease pushes; done means simultaneous releases no longer cancel unrelated queued runs or leave bumps unapplied.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, github-actions, yaml
Domain
ci-cd, devops, release
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.