Enabling 'Cancel Prebuilds on Outdated Commits' Causes Some Prebuilds to Be Marked as SYSTEM ERROR / Failed

Bug description

When enabling the 'Cancel Prebuilds on Outdated Commits' functionality on a project it can cause some prebuilds to go into a SYSTEM ERROR state and the prebuilt workspace is marked as failed. It also logs as an ERROR in the cluster - "workspace failed". This appears to mostly happen when a prebuild is longer running and a new commit is pushed to the projects repository.

While there actually does seem to be an error because the prebuild is "prematurely" stopped I don't believe we should mark this as an error or as failed. The reason why there's an error is because it was cancelled which can causes processes (like init tasks) to just exit or not run to their completion.

You'll notice in that screenshot the top prebuild is okay because I didn't make any more quick commit and pushes so the prebuild was allowed to fully run.

I believe there should be two things done here:

• Don't log this as an error in cluster logs.
• Always ensure workspaces are marked as Cancelled by the 'Cancel Prebuilds on Outdated Commits' functionality.

Steps to reproduce

1. You can fork my repository I used to reproduce this issue.
2. Setup a new project in Gitpod using this repository.
3. Within the projects Settings > General, ensure that 'Cancel Prebuilds on Outdated Commits' is checked off.
4. Open a new workspace for this project.
5. You can use the test.txt file to make modifications too. I just kept adding a couple characters, or whatever and then committed and pushed the change.
6. Modify the file again, commit and push the changes again. I repeated this 2-3 more times.
7. Observe on the prebuild tab of your project that you see a bunch of SYSTEM ERRORs.

Workspace affected

jimmybranca-gitpoddocke-0y4i1f054dq

Expected behavior

I would expect that workspaces that are cancelled when using the 'Cancel Prebuilds on Outdated Commits' are left marked as Cancelled in the UI.

Example repository

https://github.com/jimmybrancaccio/gitpod-docker-compose

Anything else?

SELECT * FROM d_b_prebuilt_workspace WHERE id = 'a669c48a-2c25-4a3b-84bc-bb3b35b5b10d';

Front conversations

Start by reproducing the issue with the linked gitpod-docker-compose repository and the documented sequence of rapid commits while prebuild cancellation is enabled. Trace the prebuild cancellation and workspace-state handling paths; done means cancelled prebuilds are shown as Cancelled rather than Failed and cancellation does not produce an error-level cluster log.