NVIDIA / NVIDIA/nvcf

byoo-otel-collector re-downloads Go SDK + collector dep graph every CI build (repo-cache eviction)

Open
#373 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

released
Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Symptom

The bazel (byoo-otel-collector) matrix job is slow on nearly every run and appears to re-download a large amount each build, even when the EC2 Buildbarn remote cache is healthy for other services.

Diagnosis

byoo re-fetches repository downloads, which the remote action cache does not cover:

  • src/compute-plane-services/byoo-otel-collector/MODULE.bazel runs go_sdk.download(version = "1.25.6") (full Go SDK, ~150 MB) as a repo rule.
  • It has two Go modules via go.work; the second (otelcol) is the full OpenTelemetry Collector contrib dependency graph (hundreds of modules) resolved through @gazelle++go_deps.
  • The EC2 Buildbarn remote cache (.github/workflows/bazel.yml) caches action outputs, not repo-rule / go_deps / SDK downloads. Those come only from the on-disk repo cache, restored by actions/cache@v4 keyed per-subtree on MODULE.bazel.lock + .bazelversion.
  • byoo has by far the largest repo-download footprint of the ~18 matrix rows. GitHub Actions cache is limited to 10 GB per repo with LRU eviction shared across all rows, so byoo's entry is very likely evicted between runs, forcing a full re-download of the SDK + Go dep graph each build.

(The ocb path is not the cause: otelcol is pre-generated and checked in via scripts/regenerate-otelcol.sh + tools/ci/check-otelcol-generated, not regenerated at build.)

Fix options (in rough order of durability)

  1. Back repo-rule downloads with the remote cache via a remote downloader / remote asset API (--experimental_remote_downloader) so the SDK + go_deps fetches are cache-served and not subject to GH 10 GB eviction. Requires the Buildbarn to expose the remote asset service; confirm availability.
  2. Give byoo a dedicated, isolated actions/cache scope (separate key namespace / larger retention) so its large repo cache is not evicted by the other rows sharing the 10 GB pool.
  3. Pre-seed a persistent repo cache (e.g. a warmed cache artifact refreshed on main) that byoo restores from.

Verify

Confirm the hypothesis from a byoo CI run log: check the actions/cache restore step for a cache miss and whether Bazel logs Go SDK + external repo fetches. Then measure cold vs warm wall-clock after the chosen fix.

Relates to the EC2 Buildbarn remote-cache work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/compute-plane-services/byoo-otel-collector/MODULE.bazel and .github/workflows/bazel.yml, then inspect a byoo CI log for the actions/cache restore result and Bazel repository fetches. Compare cold and warm wall-clock times after selecting a cache approach; done means SDK and Go dependency downloads are no longer repeatedly fetched despite the shared GitHub Actions cache limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go
Domain
build-system, ci-cd, cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.