Epic: Rewrite `ws-manager` as a Kubernetes controller

Summary

ws-manager is currently a very complex piece of software. This complexity reduces our throughput and increases the chance of introducing errors. That can be mitigated by rewriting it as a Kubernetes controller.

Context

When we built ws-manager more then three years ago, kubebuilder wasn't a thing and writing Kubernetes controllers was hard(er). Also, we didn't know too much about Kubernetes at the time.

Value

• Decreased technical complexity, which means increased throughput, faster team onboarding, and less chance of introducing errors
• Unlocking the possibility of having workspaces that are not pods, but VMs

Acceptance criteria

• Code complexity is significantly reduced
• Code coverage of new implementation is above 70%

Tasks

PoC branch is merged to main

• Introduce compatibility mode in ws-proxy to keep working with ws-manager-mk1
• #13335

Basic workspace operation parity

• #13336
• Remove classic content init in favour of PVC
• Fix CR removal on workspace stop
• #13338

ws-daemon and ws-manager don’t require gRPC anymore

• Store user env vars in secret and reference that secret from CR (OTS removal)
• Replace ws-daemon gRPC call with CR interaction

Prebuild Operation Parity

• Make mk2 understand prebuild pod termination, incl. PVC snapshot creation
• Make ws-daemon upload logs based on CR status
• Support system-failed prebuild restarting
• Ensure correct prebuild status reporting on the gRPC API

mk2 in production for Gitpod.io

• Make mk2's metrics compatible with the current mk1 metrics