Improve Experience When Deleting a Workspace Where a Workspace Pod Doesn't Exist

Bug description

This stems from an issue reported by a user in Discord.

They had run out of disk space which caused building new images (for new workspaces) to fail. An entry for the workspace was created in MySQL, however the pod itself for the workspace was never created (because of the disk space issue). When attempting to delete the broken workspace it fails because the workspace pod is non-existent.

If the user goes back to /workspaces the broken workspace still shows there.

Steps to reproduce

1. Spin up a self-hosted instance of Gitpod. I say self-hosted because while it's possible to replicate this on Gitpod.io it would be more difficult as I imagine the disk space is quite large and would be more difficult to fill up. As well as this would cause issues for real customers.
2. Some how fill up the disk where workspace pods are located.
3. Attempt to create a new workspace. It should fail due to lack of disk space. You'll be left in a state where information about the pod exists in at least MySQL and perhaps other places, and no workspace pod.
4. Navigate to /workspaces and attempt to delete the broken workspace.
5. Observe in your web browsers console and error about being unable to delete the workspace due to the workspace pod not being able to be found.

Workspace affected

No response

Expected behavior

I would expect the deletion to fail a bit more gracefully as so not to prevent the deletion of the workspace cruft in at least MySQL. There might be other data the broken workspace leaves behind besides what's in MySQL.

Example repository

No response

Anything else?

Ideally we should fail more gracefully here and attempt to delete the workspace pod, if it doesn't exist perhaps notify the user in some sort of way in the UI (a toast notification or alert component that says something like "while deleting the workspace we had an issue removing the pod because it doesn't exist, but we still deleted the workspace anyways"), but don't fail completely preventing the removal of the workspace cruft in the database as well.

Either that or we perform a disk space check before attempting to create a new workspace to ensure there's enough disk space for the new workspace.

Of note, I am not sure if this is really a bug or feature request since there's really no defects in the code, it's working as intended, but "intended" in this case could use some improvement 😝

Start at the /workspaces deletion flow and trace how workspace pod removal and MySQL cleanup are handled when the pod is absent. Reproduce the broken-workspace scenario if possible; done means deleting the workspace no longer fails entirely when its pod is missing, and the remaining workspace data is removed.