envoyproxy / envoyproxy/envoy

Set up proper caching for mobile CI

Open
#26,970 6 comments 0 reactions 0 assignees View on GitHub
area/ci area/test flakes enhancement help wanted mobile no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

Recently, we have added a bunch of caching to Envoy's CI which has both sped things up and fixed (or seriously mittigated) a load of persistent flakes that have been endemic (optimistically i would say until ~now)

Essentially there are a few things worth caching:

- docker build image
- bazel directories (repository_cache, external, .cache)

Regarding the docker image it is huge - to make it work faster than pulling from dockerhub in Envoy's CI i have had to load it using tmpfs - any other way would be significantly slower

I held off doing this for a long time because dockerhub _can_ be faster - but now that we have done this it has been a **huge** improvement to both performance and reliability. Anecdotally i would say that the azp cache gets faster the more you use it whereas the reverse seems to be true with the dockerhub cache (with some caveats)

Likewise with the bazel directories, a huge number of flakes occur just because of transient network issues, and caching these mostly eliminates this problem

While working on this for Envoy one thing i noticed was that incorrect use of bazel was creating multiple environments, fixing that was a small but sure optimization. Envoy now dumps pretty good fs info at the end of the build, and this is invaluable to understanding what bazel is doing and why it goes wrong - i strongly suggest we do the same for the mobile ci

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.