byoo-otel-collector re-downloads Go SDK + collector dep graph every CI build (repo-cache eviction)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Symptom
The bazel (byoo-otel-collector) matrix job is slow on nearly every run and appears to re-download a large amount each build, even when the EC2 Buildbarn remote cache is healthy for other services.
Diagnosis
byoo re-fetches repository downloads, which the remote action cache does not cover:
src/compute-plane-services/byoo-otel-collector/MODULE.bazelrunsgo_sdk.download(version = "1.25.6")(full Go SDK, ~150 MB) as a repo rule.- It has two Go modules via
go.work; the second (otelcol) is the full OpenTelemetry Collector contrib dependency graph (hundreds of modules) resolved through@gazelle++go_deps. - The EC2 Buildbarn remote cache (
.github/workflows/bazel.yml) caches action outputs, not repo-rule /go_deps/ SDK downloads. Those come only from the on-disk repo cache, restored byactions/cache@v4keyed per-subtree onMODULE.bazel.lock+.bazelversion. - byoo has by far the largest repo-download footprint of the ~18 matrix rows. GitHub Actions cache is limited to 10 GB per repo with LRU eviction shared across all rows, so byoo's entry is very likely evicted between runs, forcing a full re-download of the SDK + Go dep graph each build.
(The ocb path is not the cause: otelcol is pre-generated and checked in via scripts/regenerate-otelcol.sh + tools/ci/check-otelcol-generated, not regenerated at build.)
Fix options (in rough order of durability)
- Back repo-rule downloads with the remote cache via a remote downloader / remote asset API (
--experimental_remote_downloader) so the SDK + go_deps fetches are cache-served and not subject to GH 10 GB eviction. Requires the Buildbarn to expose the remote asset service; confirm availability. - Give byoo a dedicated, isolated
actions/cachescope (separate key namespace / larger retention) so its large repo cache is not evicted by the other rows sharing the 10 GB pool. - Pre-seed a persistent repo cache (e.g. a warmed cache artifact refreshed on main) that byoo restores from.
Verify
Confirm the hypothesis from a byoo CI run log: check the actions/cache restore step for a cache miss and whether Bazel logs Go SDK + external repo fetches. Then measure cold vs warm wall-clock after the chosen fix.
Relates to the EC2 Buildbarn remote-cache work.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/compute-plane-services/byoo-otel-collector/MODULE.bazel and .github/workflows/bazel.yml, then inspect a byoo CI log for the actions/cache restore result and Bazel repository fetches. Compare cold and warm wall-clock times after selecting a cache approach; done means SDK and Go dependency downloads are no longer repeatedly fetched despite the shared GitHub Actions cache limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, go
- Domain
- build-system, ci-cd, cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100