roachtest: investigate preemptible vm usage and benefit

### Background
Today, we run many tests including benchmarks using spot VMs on GCE. The main assumption behind using spot is cost saving, but it also introduces toil, and puts a burden on the test selection heuristic

https://github.com/cockroachdb/cockroach/blob/9593896c300c63f83570ccb453bda96f7d8757bf/pkg/cmd/roachtest/test_runner.go#L341
```go
if (roachtestflags.Cloud == spec.GCE || (roachtestflags.Cloud == spec.AWS &&
tests[i].Benchmark)) &&
!tests[i].Suites.Contains(registry.Weekly) &&
!tests[i].IsLastFailurePreempt() &&
rand.Float64() <= 0.75 {
lopt.runnerL.PrintfCtx(ctx, "using spot VMs to run test %s", tests[i].Name)
tests[i].Cluster.UseSpotVMs = true
}
```

> Agreed but I'd point out that due to test selection, (1/3 chance for a test to be chosen every nightly) its already possible for a test to get unlucky and be skipped for an entire week. Given that we have 700+ tests, its actually pretty likely to have a handful of tests to go the entire week without a test run.

https://github.com/cockroachdb/cockroach/pull/170652

### Goal
With the above, and as test selection is being redesigned https://github.com/cockroachdb/cockroach/issues/169712, I believe it is worth doing an investigation to see
* How much cost savings using Spot instances provide us e.g.: in the worst case a test may preempt towards the end of a test run and it would need to be run the following day (or even multiple days). How does this worst case compare to just running a test on a non spot instance
* What is the rate of preemption? We may be seeing an inconsistent run rate of benchmarks due to preemption and current test selection strategy.
* Given real cost savings and rate of preemption should we still use preemptible vms?

Datadog roachtest runner logs should be helpful in doing a short term analysis (log retention is at 1 month). It would be nice to be able to look at a longer range if the effort is reasonable, otherwise 1 month should be enough to be able to draw conclusions.

Jira issue: CRDB-64151