prebuilds: Support for force pull of the latest image manifests on manually-triggered prebuilds and webhook prebuilds

About this Issue

On projects using custom workspace images either through image key in the configuration file or through their configured custom workspace Dockerfile, when Gitpod first found a workspace image, assuming container image repositories are checked by tags, are not in its local registry proxy, it'll pull from whenever the registry the image is located and then cached it aggressively.

In my case, I maintain a fork of gitpod-io/workspace-images and use Red Hat Quay Container Registry's built-in image builder (instead of using Dazzle in GitLab CI which currently I implement ShellCheck + Hadolint checks for a while) for all the images within the quay.io/gitpodified-workspace-images/* namespace and then use it on my own projects. The problem is whenever I want to update the config file, Gitpod uses the cached version of the workspace image (possibly to save bandwidth and to avoid rate limits for unauthenticated pulls like in Docker Hub but maybe not in others) and things just went into chaotic errors.

Suggestion

• Add a dropdown menu beside Run Prebuild with a boolean option called Pull latest manifest when ticked, it will pull the latest image manifest first.
• Add support for pullLatestManifest=true URL parameter on both manual prebuild URLs and webhook endpoints (e.g. https://gitpod.io/#prebuild/https://gitlab.com/gitpodify/gitpodified-workspace-images?pullLatestManifest=true and https://gitpod.io/apps/gitlab/?pullLatestManifest=true

Workarounds

Like the Gitpod team is doing, I can change the image key of it every time and wait for prebuilds to finish.

# we can get the exact manifest digest if we want, kinda like those how npm/yarn lockfiles does minus any Dockerfile-specific
# lockfiles ;(
image: quay.io/gitpodified-workspace-images/full@sha256:14abc95e25cfbef35eda9fa1272ed39a1c2177404fd41f9b50c01e45ff5bf854

# in case of images in the gitpodified-workspace-images RHQCR namespace, including the future recaptime-dev-environment image,
# we can also use per-commit tags in form of build-<short SHA>
image: quay.io/gitpodified-workspace-images/vnc:build-c0a089c

Currently, there's no prefix for branches yet due to me slapped the Tag manifest with the branch or tag name box and no additional tag template in form of branch-${parsed_ref.branch} as I reproduced it below.

Start with the Run Prebuild flow, the manual prebuild URLs, and the webhook endpoints described in the issue. Trace how the image key and cached workspace image are handled, then determine the changes needed for a pullLatestManifest option; done means both manual and webhook prebuilds can request the latest image manifest.