[RFC] Optimizing CI Runner Utilization

Nobody has claimed this yet.

### Description

We are exploring optimizations for the CI workflow graph and would value your feedback on our proposed design.

Our primary strategy is to reduce overall queue times by offloading tasks that do not require specialized hardware to standard runners and skipping tasks early if results are already cached. These improvements specifically target our two main bottlenecks: bitstream generation and FPGA testing.

A known drawback of this approach is an increase in the number of skipped checks displayed in the UI.

## Bitstream Generation

* https://github.com/lowRISC/opentitan/pull/29634

For bitstream workflows, we propose moving cache-related tasks to standard Ubuntu runners. A dedicated bitstream runner will only be provisioned when a full build is necessary.

**Question:** Can we enforce cache usage for eg100 branch, since the hardware design is freezed?

## FPGA Testing

* https://github.com/lowRISC/opentitan/pull/29587

We propose decomposing FPGA testing into three stages:

* **Query:** Identify test targets.
* **Build:** Generate test artifacts.
* **Execute:** Run tests on FPGA hardware.

This separation minimizes the time FPGA runners are occupied while not actively utilizing the hardware. Furthermore, by reducing localized resource contention, this approach may mitigate OOM and timeout failures during peak usage.

Currently, the build task runs on the Vivado runner. To fully support parallel builds, we would need to scale our cloud cluster pool.

Note that this approach will result in increased network costs.

### Artifact Sharing

Artifacts must be transferred from build runners to execution runners. While post-merge builds can share artifacts via the remote cache, pull-request workflows are restricted from uploading to this bucket.

Until a remote execution solution is fully implemented, we are bridging the gap with our existing infrastructure. Our current method involves creating a custom remote cache overlay via Nginx and uploading it as a GitHub artifact.

#### Artifact Sharing Alternatives

We evaluated several alternatives, but found our current approach most effective:

* **Disk Cache:** Utilizing Bazel's `--disk_cache` flag prevented us from discarding intermediate artifacts, resulting in sizes 4x to 10x larger than our current method.
* **Remote Bucket Overlays:** While this allows sharing between parallel builds, it requires ACL solution to prevent cache poisoning.

We also tried building all artifacts within a single job to dedup actions, but this doubles total latency by eliminating the advantages of parallel execution.

Read the proposed CI workflow changes in PRs #29634 and #29587, focusing on the eg100 cache decision, the Query/Build/Execute split, and artifact sharing through Nginx and GitHub artifacts. The RFC does not define a concrete implementation target or completion criterion, so confirm the chosen design and measurable queue-time or resource goals before starting.

Written by the indexing model from the issue text.