[ci] Use improved cache indexing for bitstreams in the GCS cache
- Dominant language
- SystemVerilog
- Stars
- 3.6k
- Forks
- 1.1k
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 141
Description
### Description
This is a follow up of issue #31059, where the GitHub cache was introduced to skip FPGA bitstream rebuilds on "trivial" PR updates (i.e. updates which do not touch the bitstream sources and hence can safely reuse the bitstream computed for a previous version of that PR).
This issue proposes to change the GCS cache to also store the "FuseSoC SHA" (i.e. the key used to index the GH cache, see below). The [get-bitstream-strategy.sh](https://github.com/lowRISC/opentitan/blob/1e1dace7680251f88ab11adedd8766222f333962/ci/scripts/get-bitstream-strategy.sh) script can then be changed to use that key to check whether the bitstream of the commit closest to the tested `HEAD` can be reused for the current PR (rather than looking at the diff between the two commits and applying the wonky list of exclusions as explained below).
A suggestion is also to rename the `get-bitstream-strategy.sh` file and its output `bitstreamStrategy` to reflect the changes made. E.g. The script could be renamed to `check-bitstream-gcs-cache.sh` and its output to `cache-hit` (for consistency with GH cache action).
## Background
At the moment we have 3 caching mechanisms for bitstreams:
1. The Google Cloud Storage (GCS) bitstream cache [here](https://github.com/lowRISC/opentitan/blob/1e1dace7680251f88ab11adedd8766222f333962/ci/scripts/get-bitstream-strategy.sh), which looks at the current `HEAD`, finds the closest past bitstream in the GCS cache (at `HEAD`, or `HEAD~1`, or `HEAD~2`, ...), finds the diff between the two commits, excluding a hand-curated list of paths deemed irrelevant for bitstream computation, and establishes a "cache-hit" condition if that diff is empty.
2. The internal Bazel cache, that identifies whether Bazel already built the bitstream based on the files provided as a dependency to FuseSoC.
3. The GitHub cache, which computes a hash of all files based on a list of files obtained from FuseSoC. We call it "FuseSoC SHA" in this issue.
These three caching mechanisms use three different cache indexing functions, i.e. three different functions to associate a cache entry based on the state of the source tree. This means that a commit may fail to hit the GCS cache and still hit the GH cache, leading to unexpected behaviours. This issue aims to reduce the differences between two of the three cache indexing functions.
**Note**: The caching mechanism (1) is less "precise" than (2), which is itself less precise than (3). By this, we mean that (1) may report a cache miss when (2) may instead report a cache hit. This is because (1) ends up considering source files which are not really used for building the bitstream. A commit touching one of these files may end up causing (1) to report a miss, while the more precise (2) may instead correctly recognise that the touched file is not a dependency of the bitstream. (3) is the more "precise", meaning that it considers a reduced set of source files. However, [it was discussed](https://github.com/lowRISC/opentitan/pull/31260#discussion_r3958647907) that (3) fails to account for the actual FuseSoC command line, which is itself a dependency of the bitstream. This could lead to "surprises", where a bitstream is taken from the cache even if it should really have been rebuilt.
Contributor guide
Research direction
Start with ci/scripts/get-bitstream-strategy.sh and the caching behavior described in issues #31059 and #31260. Trace how the script finds a prior GCS bitstream and how the GitHub cache supplies the FuseSoC SHA. Done means the GCS cache stores and checks that key instead of relying on the hand-curated diff exclusions, with any naming changes applied consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, github, shell
- Domain
- build-system, ci-cd, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100