rivet-dev / rivet-dev/agentos

ci: darwin release builds are uncached — port rivet sccache/cache-mount Dockerfile pattern

Open
#1,639 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
4.6k
Forks
251
Avg merge
2d 15h
Merged PRs (30d)
34

Description

The "Build darwin binaries" jobs are the critical path for Publish npm in publish.yaml (~10.5 min per release) and rebuild ~388 crates from scratch every run.

Current state

  • docker/build/darwin.Dockerfile uses FROM ghcr.io/rivet-dev/rivet/builder-base-osxcross:0e33ceb98 — the same prebaked base rivet uses (osxcross + MacOSX 11.3 SDK + both apple Rust targets + sccache preinstalled) — but uses none of its caching affordances:
    • re-installs the rust toolchain/targets the base already has
    • no RUSTC_WRAPPER=sccache, no cache backend, no BuildKit cache mounts
    • plain docker build with no --cache-from/--cache-to
  • Measured on the 0.2.5 stable run (28801821648): base image pull ~1 min, rustup ~1 min, cargo build --release of 388 crates = 7m52s, every run, forever.
  • By contrast the linux sidecar/plugin jobs use Swatinem/rust-cache and finish in 1–5 min warm.

Fix: port the rivet consumer pattern

Rivet already solved this for the identical build in rivet-dev/rivet:

  • docker/build/darwin-{arm64,x64}.Dockerfile: RUSTC_WRAPPER=sccache + SCCACHE_WEBDAV_ENDPOINT=https://cache.depot.dev, --mount=type=secret,id=DEPOT_TOKEN,env=SCCACHE_WEBDAV_TOKEN with a graceful health-check fallback that disables sccache when the token/backend is unavailable, plus --mount=type=cache for cargo registry/git/target.
  • Base image build/push lives in docker/builder-base/osxcross.Dockerfile + scripts/docker-builder-base/build-push.sh.

Port notes for agentos:

  • agentos builds on GitHub-hosted runners (not Depot builders), so BuildKit cache mounts alone will not persist across runs — the cross-run win is the sccache remote backend. Either:
    • (a) add a DEPOT_TOKEN secret and reuse rivet's Depot cache config verbatim (preferred: byte-for-byte port of a production pattern), or
    • (b) use sccache's native GHA backend (SCCACHE_GHA_ENABLED=on).
  • Drop the redundant rustup toolchain/target install from darwin.Dockerfile (already in the base).
  • Pass the secret through in the build-sidecar-darwin job in .github/workflows/publish.yaml (docker buildx build --secret id=DEPOT_TOKEN).

Expected: darwin job drops from ~10 min to ~2–3 min warm, cutting release npm-publish latency roughly in half.

Related smaller wins (same investigation, can be separate PRs)

  • Publish crates.io job: give it its own concurrency group (or workflow) so a slow crates tail (crates.io new-crate rate-limit backoff can sleep for many minutes) doesn't queue the next release's npm path behind it (concurrency: publish-${{ github.ref }}, cancel-in-progress: false caused the 0.2.6 dispatch to sit pending behind 0.2.5's crates job). Also surface the "rate limited, retrying at " sleep as a ::notice:: so the job doesn't look hung.
  • Release rust-cache eviction: linux jobs key caches by <target>-<trigger>, so -release caches are only warmed by prior release runs and get LRU-evicted during quiet stretches while -preview stays warm; consider a shared key or a scheduled warm job.
  • WASM Commands (~8.6 min): cache packages/runtime-core/commands keyed on hashFiles('registry/native/**') and skip make on hit.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare docker/build/darwin.Dockerfile with rivet's docker/build/darwin-{arm64,x64}.Dockerfile, then inspect the build-sidecar-darwin job in .github/workflows/publish.yaml and the named builder-base files. Verify the resulting warm darwin release build uses the remote cache, avoids redundant rustup setup, and falls back gracefully when the cache secret is unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, rust
Domain
build-system, ci-cd, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.