[ci] CI critical path tracking issue

@charles-at-lowrisc is already working on this.

Since Nov 16, 2023.

### Description

This issue is for tracking work towards optimising the critical path of our CI runs to reduce the start-to-end time.

### Data

These dashboards are useful for analysing the critical path:

* [Job Explorer](https://dashboards.lowrisc.org/grafana/d/d8301520-a258-4cef-8b9e-2b0c1ac4fccc/job-explorer?orgId=1) - enter the ID of an Azure pipelines run (`?buildid=` in [a run's URL](https://dev.azure.com/lowrisc/opentitan/_build/results?buildId=134237&view=results)) to see its Gantt chart.
* [Total CI Duration](https://dashboards.lowrisc.org/grafana/d/aa6a3c5e-d323-464f-9ff0-1ccb6221d2fd/ci-duration?orgId=1) - time from creation to finish of all successful master-branch CI runs.
* [CI Dashboard](https://dashboards.lowrisc.org/grafana/d/ebe2c4c0-bb38-47ab-aac0-c04c9956898d/ci-dashboard?orgId=1) - tracks the durations of individual jobs and tests.

You can view these dashboards by signing in with an OpenTitan account. Apologies if you're reading this and don't have one.

### Analysis

Typical CI runs looked like this a few months ago (3+ hours):


Among these jobs are:

* Bitstream builds which take about an hour each and run in parallel to one another.
* SW Build & Test which takes by far the longest at around 2 hours.
* FPGA tests over on the right, which depend on both bitstreams and SWB&T.

### Progress

Some improvements so far:

1. #19650
* Improved dependency tracking has *greatly* reduced the SWB&T job to ~50m.
2. #20307
* Investigations into the bitstream cache give us more cache hits, avoiding 1-hour rebuilds.
3. #19577
* SWB&T was split into separate build and test jobs so that FPGA tests can depend on just the build and not the tests. Saves ~15 minutes with a bitstream cache hit.

Typical CI runs now look like this (1h 34m):


### Further work

The critical path is now:

```
Quick lint --+--> bitstreams (if not cached) --+--> CW310 ROM tests
\ /
'--> SW build (if cached) ----'
```

Work is ongoing to improve bitstream cache hits further, and I suspect the SW build job isn't using as many cached artifacts as it could, but both of these need investigation.

The Quick Lint job takes ~10 minutes and contains a 5 minute documentation build. I'm moving this to Slow Lints in #20339.