Epic: Spot VMs for headless workspaces

Summary

Currently we are using regular VMs for our headless node pool (prebuilds).
More efficient would be to use spot VMs (which are 60-90% cheaper). We can have both cheaper and faster VMs if we use spot VMs, since we can use bigger VMs (faster prebuilds) and still save on cost.
Spot VMs can be shutdown at any moment though (in GKE google gives several minutes to shutdown VM safely, need to check if same is applicable to GCP VMs), so we need to make sure that if prebuild was running, we can restart it on a different VM instead. In general those shutdowns shouldn't happen frequently.

Another option is to have bare metal cluster (rented outside of GCP). But I think we can pursue both options, have SpotVMs for regions where we might not be able to get bare metal cluster.

Context

Slack thread: https://gitpod.slack.com/archives/C02F19UUW6S/p1643645109149259

Value

Cheaper cost for gitpod, faster prebuilds for users. Win\win.

Potential savings on the headless cost:

• Maximum: 72.76%
• Current: 57.39%
• Minimum: 50.34%

(internal) Additional calculations

Acceptance criteria

Prebuilds should be restart-able in the event it did not finish.
Headless pool should be working without issues in node pool consisting of Spot VMs only.

Measurement

Hypothesis

In scope

See if we can include image builds. Additionally, if the retry mechanism can live on workspace side, and not have to be triggered by webapp, that would help.

Out of scope

Complexities

No files, tests, or entry points are named. Start by reviewing the Spot VM and prebuild behavior described here, including GKE shutdown handling and restart requirements; done means prebuilds can restart after interruption and the headless pool runs entirely on Spot VMs.