NVIDIA / NVIDIA/nvcf

ci: five workflows run on every push with no path filter

Open
#977 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Problem

Seven workflows fire on every push. Five of them have no path filter at all:

build-test.yml            runs on every push
license-dependencies.yml  runs on every push
secret-scan.yml           runs on every push
byoo-otel-collector.yml   runs on every push
bazel.yml                 runs on every push
codeql.yml                paths-filtered
openbao-jwt-plugin.yml    paths-filtered

The volume this produces is what pushes the tail of runner wait times out. Measured 2026-08-18:

day            n   median    p90     max   over 60s
Fri 08-14     32       2s     3s      5s      0%
Mon 08-17     48       1s    59s    147s      8%
Tue 08-18     37       0s    43s    606s      8%

Median is fine; the worst case went from 5 seconds on Friday to 10 minutes. At the time of measurement 15 runs were queued, the oldest for 35 minutes. 881 runs that day, with the 100 most recent all inside a single hour, against 770 for all of Friday.

The 10 minute waits cluster inside single runs (dependency licenses, go-lib codegen, docs, GitHub release helper, license headers, generated dependency docs and Secret Scan all waited about 610s together), which is a concurrency ceiling rather than a slow runner. GitHub reported all systems operational with no incidents.

One branch, nvsnap/e2e-restore-guards, produced 25 runs in a day by itself across about four pushes.

Proposal

Extend the pattern from NVIDIA/nvcf#949 to the remaining always-on workflows: keep the job running so it always reports and branch protection is untouched, and gate only the expensive steps on a scope decision computed from the merge base.

The two constraints that made #949 non-trivial apply here too and should not be re-derived:

  • A skipped required check never reports, so on.pull_request.paths blocks the merge unless a companion no-op job is added. Gating steps rather than the job avoids this entirely.
  • The input set is usually wider than it looks. For generated dependency docs the Java runtime inventory is built through Bazel, so a Java component's BUILD.bazel counts even though no manifest changed. Each workflow needs its real input set worked out, not guessed, and the decision must fail closed.

bazel.yml is already change-aware and should stay as is; #974 narrows its one remaining over-broad edge.

Order

build-test.yml and byoo-otel-collector.yml first: they are the ones most likely to be scopeable to a subtree. secret-scan.yml should probably stay unscoped, since scanning only changed paths defeats the point of scanning.

Each one needs its own change with its own tests, in the shape of tools/ci/dependency-docs-scope and its behavioral test, so the fail-closed behaviour is verifiable rather than asserted.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing tools/ci/dependency-docs-scope implementation and its behavioral test, then inspect build-test.yml and byoo-otel-collector.yml as the proposed first workflows. Work out each workflow's real input set and verify that required checks still report while expensive steps are gated. Done means fail-closed scope decisions with separate behavioral coverage for each workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
build-system, ci-cd
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.