Resolve Issue When Using a Self-Hosted GitLab Instance for the Container Registry on a Self-Hosted Gitpod Instance

Bug description

When attempting to use the container registry of a self-hosted GitLab instance the following error is displayed in the Gitpod UI when bringing up a new workspace:

Request createWorkspace failed with message: 13 INTERNAL: cannot resolve workspace image: hostname required
Unknown Error: { "code": -32603 }

This message seems a bit misleading if I am to also believe what is shown in the image-builder-mk3 logs:

Mon, Mar 14 2022 8:19:20 am     {"host":"registry.gitpodbread.com","level":"debug","message":"do request","request.header.accept":"application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*","request.header.user-agent":"containerd/1.5.5+unknown","request.method":"HEAD","severity":"DEBUG","time":"2022-03-14T13:19:20Z","url":"https://registry.gitpodbread.com/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d"}

Mon, Mar 14 2022 8:19:20 am	{"host":"registry.gitpodbread.com","level":"debug","message":"fetch response received","response.header.content-length":"159","response.header.content-type":"application/json","response.header.date":"Mon, 14 Mar 2022 13:19:20 GMT","response.header.docker-distribution-api-version":"registry/2.0","response.header.server":"nginx","response.header.www-authenticate":"Bearer realm=\"https://gitlab.gitpodbread.com/jwt/auth\",service=\"container_registry\",scope=\"repository:workspace-images:pull\"","response.header.x-content-type-options":"nosniff","response.status":"401 Unauthorized","severity":"DEBUG","time":"2022-03-14T13:19:20Z","url":"https://registry.gitpodbread.com/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d"}

Mon, Mar 14 2022 8:19:20 am	{"header":"Bearer realm=\"https://gitlab.gitpodbread.com/jwt/auth\",service=\"container_registry\",scope=\"repository:workspace-images:pull\"","host":"registry.gitpodbread.com","level":"debug","message":"Unauthorized","severity":"DEBUG","time":"2022-03-14T13:19:20Z"}

Mon, Mar 14 2022 8:19:20 am	{"host":"registry.gitpodbread.com","level":"debug","message":"do request","request.header.accept":"application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*","request.header.user-agent":"containerd/1.5.5+unknown","request.method":"HEAD","severity":"DEBUG","time":"2022-03-14T13:19:20Z","url":"https://registry.gitpodbread.com/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d"}

Mon, Mar 14 2022 8:19:20 am	{"host":"registry.gitpodbread.com","level":"debug","message":"fetch response received","response.header.content-length":"159","response.header.content-type":"application/json","response.header.date":"Mon, 14 Mar 2022 13:19:20 GMT","response.header.docker-distribution-api-version":"registry/2.0","response.header.server":"nginx","response.header.www-authenticate":"Bearer realm=\"https://gitlab.gitpodbread.com/jwt/auth\",service=\"container_registry\",scope=\"repository:workspace-images:pull\",error=\"insufficient_scope\"","response.header.x-content-type-options":"nosniff","response.status":"401 Unauthorized","severity":"DEBUG","time":"2022-03-14T13:19:20Z","url":"https://registry.gitpodbread.com/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d"}

Mon, Mar 14 2022 8:19:20 am	{"header":"Bearer realm=\"https://gitlab.gitpodbread.com/jwt/auth\",service=\"container_registry\",scope=\"repository:workspace-images:pull\",error=\"insufficient_scope\"","host":"registry.gitpodbread.com","level":"debug","message":"Unauthorized","severity":"DEBUG","time":"2022-03-14T13:19:20Z"}

Mon, Mar 14 2022 8:19:20 am	{"error":"authorization failed\ngithub.com/containerd/containerd/remotes/docker.init\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:46\nruntime.doInit\n\truntime/proc.go:6498\nruntime.doInit\n\truntime/proc.go:6475\nruntime.doInit\n\truntime/proc.go:6475\nruntime.doInit\n\truntime/proc.go:6475\nruntime.doInit\n\truntime/proc.go:6475\nruntime.main\n\truntime/proc.go:238\nruntime.goexit\n\truntime/asm_amd64.s:1581\nserver message: insufficient_scope\ngithub.com/containerd/containerd/remotes/docker.invalidAuthorization\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/authorizer.go:322\ngithub.com/containerd/containerd/remotes/docker.(*dockerAuthorizer).AddResponses\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/authorizer.go:136\ngithub.com/containerd/containerd/remotes/docker.(*request).retryRequest\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:603\ngithub.com/containerd/containerd/remotes/docker.(*request).doWithRetries\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:582\ngithub.com/containerd/containerd/remotes/docker.(*request).doWithRetries\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:589\ngithub.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:280\ngithub.com/gitpod-io/gitpod/image-builder/pkg/resolve.(*StandaloneRefResolver).Resolve\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/resolve/resolve.go:98\ngithub.com/gitpod-io/gitpod/image-builder/pkg/resolve.(*PrecachingRefResolver).Resolve\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/resolve/resolve.go:245\ngithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator.(*Orchestrator).checkImageExists\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator/orchestrator.go:513\ngithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator.(*Orchestrator).ResolveWorkspaceImage\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator/orchestrator.go:185\ngithub.com/gitpod-io/gitpod/image-builder/api._ImageBuilder_ResolveWorkspaceImage_Handler.func1\n\tgithub.com/gitpod-io/gitpod/image-builder/api@v0.0.0-00010101000000-000000000000/imgbuilder_grpc.pb.go:218\ngithub.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/tracing/opentracing/server_interceptors.go:38\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/gitpod-io/gitpod/image-builder/api._ImageBuilder_ResolveWorkspaceImage_Handler\n\tgithub.com/gitpod-io/gitpod/image-builder/api@v0.0.0-00010101000000-000000000000/imgbuilder_grpc.pb.go:220\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\tgoogle.golang.org/grpc@v1.39.1/server.go:1293\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.39.1/server.go:1618\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\tgoogle.golang.org/grpc@v1.39.1/server.go:941\nruntime.goexit\n\truntime/asm_amd64.s:1581\npull access denied, repository does not exist or may require authorization\ngithub.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve\n\tgithub.com/containerd/containerd@v1.5.5/remotes/docker/resolver.go:283\ngithub.com/gitpod-io/gitpod/image-builder/pkg/resolve.(*StandaloneRefResolver).Resolve\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/resolve/resolve.go:98\ngithub.com/gitpod-io/gitpod/image-builder/pkg/resolve.(*PrecachingRefResolver).Resolve\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/resolve/resolve.go:245\ngithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator.(*Orchestrator).checkImageExists\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator/orchestrator.go:513\ngithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator.(*Orchestrator).ResolveWorkspaceImage\n\tgithub.com/gitpod-io/gitpod/image-builder/pkg/orchestrator/orchestrator.go:185\ngithub.com/gitpod-io/gitpod/image-builder/api._ImageBuilder_ResolveWorkspaceImage_Handler.func1\n\tgithub.com/gitpod-io/gitpod/image-builder/api@v0.0.0-00010101000000-000000000000/imgbuilder_grpc.pb.go:218\ngithub.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/tracing/opentracing/server_interceptors.go:38\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\tgithub.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/gitpod-io/gitpod/image-builder/api._ImageBuilder_ResolveWorkspaceImage_Handler\n\tgithub.com/gitpod-io/gitpod/image-builder/api@v0.0.0-00010101000000-000000000000/imgbuilder_grpc.pb.go:220\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\tgoogle.golang.org/grpc@v1.39.1/server.go:1293\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.39.1/server.go:1618\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\tgoogle.golang.org/grpc@v1.39.1/server.go:941\nruntime.goexit\n\truntime/asm_amd64.s:1581","host":"registry.gitpodbread.com","level":"info","message":"trying next host","severity":"INFO","time":"2022-03-14T13:19:20Z"}

As you can see it looks like an authorization error. I have tried setting up the container-registry-token using my GitLab administrator account to create a personal access token as well as my "normal" user account but both cause the same error message as seen above. When I create the PAT I create it with the api, read_registry and write_registry scopes.

Here's some logs from GitLab (/var/log/gitlab/registry/current - 0.0.0.0 indicates my home public IP):

2022-03-14_13:46:49.89829 time="2022-03-14T08:46:49.898-05:00" level=debug msg="authorizing request" correlation_id=01FY4AWY4WN1XVK62G2ME885SD go_version=go1.17.6 root_repo=workspace-images vars_name=workspace-images vars_reference=68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d

2022-03-14_13:46:49.89923 time="2022-03-14T08:46:49.899-05:00" level=warning msg="error authorizing context: authorization token required" correlation_id=01FY4AWY4WN1XVK62G2ME885SD go_version=go1.17.6 root_repo=workspace-images vars_name=workspace-images vars_reference=68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d

2022-03-14_13:46:49.89940 {"content_type":"application/json","correlation_id":"01FY4AWY4WN1XVK62G2ME885SD","duration_ms":14,"host":"registry.gitpodbread.com","level":"info","method":"HEAD","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:44830","remote_ip":"0.0.0.0","status":401,"system":"http","time":"2022-03-14T08:46:49.899-05:00","ttfb_ms":14,"uri":"/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d","user_agent":"containerd/1.5.5+unknown","written_bytes":159}

2022-03-14_13:46:50.08123 time="2022-03-14T08:46:50.081-05:00" level=debug msg="authorizing request" correlation_id=01FY4AWYAX5EBYH5YQ9GWQYHTX go_version=go1.17.6 root_repo=workspace-images vars_name=workspace-images vars_reference=68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d

2022-03-14_13:46:50.08229 time="2022-03-14T08:46:50.082-05:00" level=warning msg="error authorizing context: insufficient scope" correlation_id=01FY4AWYAX5EBYH5YQ9GWQYHTX go_version=go1.17.6 root_repo=workspace-images vars_name=workspace-images vars_reference=68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d

2022-03-14_13:46:50.08242 {"content_type":"application/json","correlation_id":"01FY4AWYAX5EBYH5YQ9GWQYHTX","duration_ms":4,"host":"registry.gitpodbread.com","level":"info","method":"HEAD","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:44832","remote_ip":"0.0.0.0","status":401,"system":"http","time":"2022-03-14T08:46:50.082-05:00","ttfb_ms":4,"uri":"/v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d","user_agent":"containerd/1.5.5+unknown","written_bytes":159}

And from /var/log/gitlab/nginx/gitlab_registry_access.log:

0.0.0.0 - - [14/Mar/2022:08:19:20 -0500] "HEAD /v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d HTTP/2.0" 401 0 "" "containerd/1.5.5+unknown" -
0.0.0.0 - - [14/Mar/2022:08:19:20 -0500] "HEAD /v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d HTTP/2.0" 401 0 "" "containerd/1.5.5+unknown" -
0.0.0.0 - - [14/Mar/2022:08:46:49 -0500] "HEAD /v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d HTTP/2.0" 401 0 "" "containerd/1.5.5+unknown" -
0.0.0.0 - - [14/Mar/2022:08:46:50 -0500] "HEAD /v2/workspace-images/manifests/68f415049cf6f3028c3427028d3699e5cc2ab3a915c4ddc623c4ec5d3e21369d HTTP/2.0" 401 0 "" "containerd/1.5.5+unknown" -

I also noticed in the messages from the image-builder-mk3 pod - pull access denied, repository does not exist or may require authorization so I am wondering if something else needs to be done when using GitLab as a container registry? Does a group, or project also need to be created before hand? For what it's worth, I do have a group called 'workspace-images' to which I have pushed the workspace-full:latest image to.

I am running my GitLab Container Registry instance under it's own domain (main GitLab is at gitlab.gitpodbread.com, registry is at registry.gitpodbread.com). Related gitlab.rb config:

registry_external_url 'https://registry.gitpodbread.com'
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.gitpodbread.com"
registry['log_level'] = "debug"
registry_nginx['enable'] = true
registry_nginx['proxy_set_headers'] = {
    "Host" => "$http_host",
    "X-Real-IP" => "$remote_addr",
    "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
    "X-Forwarded-Proto" => "https",
    "X-Forwarded-Ssl" => "on"
}

Gitpod configuration - gitpod.config.yaml:

apiVersion: v1
authProviders:
  - kind: secret
    name: gitlab
blockNewUsers:
  enabled: false
  passlist: []
certificate:
  kind: secret
  name: https-certificates
containerRegistry:
  inCluster: false
  external:
    url: registry.gitpodbread.com
    certificate:
      kind: secret
      name: container-registry-token
database:
  inCluster: true
disableDefinitelyGp: false
domain: gitpodbread.com
kind: Full
metadata:
  region: local
objectStorage:
  inCluster: true
observability:
  logLevel: debug
openVSX:
  url: https://open-vsx.org
repository: eu.gcr.io/gitpod-core-dev/build
workspace:
  resources:
    requests:
      cpu: "1"
      memory: 4Gi
  runtime:
    containerdRuntimeDir: /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io
    containerdSocket: /run/k3s/containerd/containerd.sock
    fsShiftMethod: fuse

If any more of my setups settings or information can be provided to further troubleshoot this please let me know! cc. @MrSimonEmms

Steps to reproduce

1. Spin up a self-hosted GitLab instance. Ideally it is publicly accessible, or at least accessible by Let's Encrypt so TLS certificates can be fetched and applied.
2. Create a normal user account and a PAT under that account with the api, read_registry and write_registry scopes.
3. Create a docker-registry secret in Kubernetes with that information.
4. Spin up a Gitpod instance and set it to use your GitLab Container Registry.
5. Attempt to create a new workspace, observe the error 'cannot resolve workspace image: hostname required'.

Workspace affected

No response

Expected behavior

I would expect that Gitpod be able to utilize the GitLab Container Registry for itself.

Example repository

No response

Anything else?

No response

Front conversations

Start with image-builder/pkg/resolve/resolve.go, especially StandaloneRefResolver.Resolve, and image-builder/pkg/orchestrator/orchestrator.go at checkImageExists and ResolveWorkspaceImage; compare the resolver's registry authentication flow with the GitLab and image-builder logs in this issue. Reproduce the self-hosted GitLab registry setup and trace the failure beyond the misleading hostname error. Done means a workspace image resolves successfully or the UI reports the actual authorization failure.