[image-builder] image build sometimes fails due to `MountVolume.SetUp failed for volume "gitpod-ca-certificate"`

Bug description

The error was:

MountVolume.SetUp failed for volume "gitpod-ca-certificate" : failed to sync secret cache: timed out waiting for the condition

As a user, I saw:

Logs

Steps to reproduce

n/a

the error is rare and intermittent

Workspace affected

n/a

Expected behavior

We should be able to mount the gitpod CA cert if the workspace container is in a good state, do the image build, and then start a workspace from it.

If we cannot mount the cert, we won't be able to pull images through registry-facade for the imagebuild workspace, and we should log the state of the pod to better understand why.

Example repository

No response

Anything else?

We might need to wait longer when creating a workspace pod, before trying to use its underlying mounts that are dependent on a secret. Example where we had to wait longer for a secret that is created at runtime. In this case, we're unable to mount a secret that was defined at install time, and then owned by cert-manager.

Start with the linked logs and the image-builder workspace pod creation flow, then inspect install/installer/pkg/common/constants.go and compare the approach in PR #15378. Determine why the gitpod-ca-certificate secret mount times out, ensure pod state is logged when mounting fails, and verify that image builds can proceed when the workspace container is healthy.