roachtest: roachtest GCE nightly CI has been timing out consistently

## Summary
We are constrained on both CI execution time and GCE Quota. While there are paths forward on just the CI execution time (e.g. splitting benchmarks into their own CI job), the root issue is GCE VM availability. For GCE availability, when release branches are being targeted by CI we further stress GCE resources.

## Why has CI runtime increased?
### Test Selection
There is ongoing test selection work, but execution times have been increasing while using the existing test selection heuristic so there must be other factors.

### Number if Roachtest tests increasing over time
[GCE Master](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/?branch=%3Cdefault%3E&mode=builds&page=2)
* Over the last few months, we've seen the number of roachtests increase, but not substantially

```text
# April 15
[06:22:48 ]selective Test enabled
[06:22:52 ]199 selected out of 566 successful tests.
[06:22:52 ]411 out of 689 tests selected for the run!

# June 14
[06:01:42 ]selective Test enabled
[06:01:44 ]179 selected out of 510 successful tests.
[06:01:44 ]479 out of 713 tests selected for the run!
```

Previously, [in this change](https://cockroachlabs.slack.com/archives/C023S0V4YEB/p1776977242525669) which increased the number of roachtests (I believe it was >100 new tests, couldn't find a note of that), we saw nightly execution times increase, but for an increase of ~20, that seems less likely to be the root cause of the recent timeouts. Although even if it isn't, the number of roachtests will increase over time so scaling up is a problem that will need to be solved.

### PR that introduces Retry for Benchmarks that have been preempted #170652
https://github.com/cockroachdb/cockroach/pull/170652
PR Slack Context: https://cockroachlabs.slack.com/archives/C023S0V4YEB/p1779285186391189
TE Thread https://cockroachlabs.slack.com/archives/C01UDP33CCS/p1781535189785499

Example:
[tc](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21385819?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&expandBuildTestsSection=false&hideProblemsFromDependencies=false&expandBuildChangesSection=true) 12412 [dd 4 retries](https://us5.datadoghq.com/logs?query=service%3Aroachtest%20%22requeueing%22%20%40build_number%3A12412&agg_m=count&agg_m_source=base&agg_q=version&agg_t=count&cols=host%2Cservice&flat_group_bys=true&fromUser=true&messageDisplay=inline&refresh_mode=sliding&sort_m=count&sort_t=count&storage=flex_tier&stream_sort=desc&top_n=10&top_o=top&viz=stream&x_missing=true&from_ts=1778956368100&to_ts=1781548368100&live=true), this build finished in 23 hrs
[tc 12400 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyGceBazel/21385238?hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandBuildTestsSection=false&expandBuildDeploymentsSection=false&expandBuildProblemsSection=true&expandBuildChangesSection=true), [dd 9 retries](https://us5.datadoghq.com/logs?query=service%3Aroachtest%20%22requeueing%22%20%40build_number%3A12400&agg_m=count&agg_m_source=base&agg_q=version&agg_t=count&cols=host%2Cservice&flat_group_bys=true&messageDisplay=inline&refresh_mode=sliding&sort_m=count&sort_t=count&storage=flex_tier&stream_sort=desc&top_n=10&top_o=top&viz=stream&x_missing=true&from_ts=1778956368100&to_ts=1781548368100&live=true), this build timed out

Given the number of occurrences of this retry mechanism being hit in datadog, and the fact that we are experiencing timeouts only in `master` and this change was only merged to `master`, I have high confidence that this is contributing to the issue. With current observability limitations, we can't conclude that this is the sole reason for the increase in the nightly execution time.

## Potential Solutions For GCE VM availability
### Use MIGs to wait for vm availability
### Stagger CI Start Times
Currently start times are ~12am ET. If we were to say start half of CI at 12am and the other half at 12pm we could reduce the peak resource usage. This assumes CI jobs finish at ~12 hours which currently isn't the case, but the test selection heuristic could be modified to aim for a 12 hour execution time.
### Use Additional Regions
Some tests may depend on latency between the test runner and the cluster. The runner could also be in a different region as well though.

## Improving Observability into Roachtest (the harness)'s Overall Duration
### Gannt Chart
Visualization, can compare based TC runs against each other.
Questions to answer:
* Where do we spend the most time? Cluster creation? Test execution?
## Schedules
This is a longer term effort / solution.
Given a list of constraints i.e. provisionable CPUs, execution time, etc. is there a schedule of test executions that minimize execution time?
An example of a schedule is where we choose to run smaller tests first.
Similar to the [bin packing problem](https://en.wikipedia.org/wiki/Bin_packing_problem).
### Cluster Reuse Strategy
Can cluster reuse strategy be adjusted or used more aggressively to see positive outcomes?

## Schedule Simulations
To help validate schedules, we can build a simulation tool for roachtest. It's input would be an output of the test selection heuristic.

## Benchmark Quality
Are all benchmarks equally as important? If not, could introducing a priority help prioritize certain benchmarks and deprioritize others reducing the number of total runs?

## Immediate steps
### Revert #170652
### Use non preemptible VMs for Benchmarks
Less preferred because of lack of evidence that preemption is the root cause of the execution time increase, and cost implications.

Jira issue: CRDB-64791