Workspace Failing to Start Due To 'metadata.annotations' Too Long

Bug description

I received a support ticket today where a customer was unable to start their workspace. After digging into the logs it seems the workspace cannot start due to metadata.annotations being too long.

cannot mark workspace bf938941-a5a5-43e1-aad9-a1af9836c11a with +gitpod/explicitFail:
    github.com/gitpod-io/gitpod/ws-manager/pkg/manager.(*Manager).markWorkspace
        github.com/gitpod-io/gitpod/ws-manager/pkg/manager/annotations.go:139
  - Pod "ws-bf938941-a5a5-43e1-aad9-a1af9836c11a" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

Steps to reproduce

This may be a little difficult, but what I would suggest:

1. Create a workspace but then stop it.
2. Some how manually manipulate the metadata.annotations of the pod so that it's more than 262144 bytes.
3. Attempt to spin up the workspace again.
4. Observe in logs that you get an error about the metadata.annotations being too long.

Workspace affected

No response

Expected behavior

I am not sure what error message is being displayed to the customer but I can attempt to find this out within the Front conversation.

Depending on the error message or error displayed we should provide more information on why that happened and the next steps a customer can take.

Example repository

No response

Anything else?

No response

Front conversations