NVIDIA / NVIDIA/nvcf

Root Bazel build is cold on every PR/merge (~626s, 0 cache hits)

Open
#283 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Problem

A PR or merge that builds the root Bazel matrix row compiles the full
~7,000-target closure. Measured cold: ~626s elapsed, 5,172 local actions,
0 remote-cache hits.

Root cause: the root row only ran on main when a ROOT_GLOB path changed
(roughly 1 merge in 10), so the root closure was almost never uploaded to the
remote cache. Frequently-touched rows like nvca warm naturally (~86% hits);
root does not. The remote cache box has ample headroom (rings ~1% used), so a
warm write would persist; it simply is rarely written.

Plan

  1. Warm root on every merge to main so the full closure is uploaded each
    merge; PRs and per-service rows then read the warm shared closure.
    (PR: perf(ci): warm root bazel cache on main; rename matrix row to root)
  2. Follow-up: build only the targets a PR's changed files affect
    (target-determinator), so a PR touching a small part of src/libraries/
    no longer rebuilds the whole root closure even on a cold cache.

Acceptance

  • After a merge to main, the root row logs upload=true and uploads the
    closure.
  • A subsequent PR touching the shared closure shows a high remote-cache hit
    ratio and a large wall-clock drop versus the ~626s cold baseline.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the CI configuration for the root Bazel matrix row and inspect the merge-to-main path described in the issue. Verify that the root row logs upload=true after a merge, then compare a subsequent shared-closure PR's remote-cache hits and wall-clock time with the ~626s cold baseline.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, ci-cd, performance
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.