containerd / containerd/nerdctl
nerdctl pull from private registry mirror is missing Authorization header
- Dominant language
- Go
- Stars
- 10.4k
- Forks
- 826
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 44
Description
### Description
I tried to use nerdctl with [registry mirrors](https://github.com/containerd/nerdctl/issues/668) served by private harbor, but I got `401 Unauthorized` error.
After some investigation, I found that when using registry mirrors, nerdctl doesn't pass the `Authorization: Basic xxxx` header to `/service/token` endpoints, causing the final `401 Unauthorized` error.
### Steps to reproduce the issue
1. configure containerd to use registry mirrors:
/etc/containerd/config.toml
```toml
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
```
/etc/containerd/certs.d/gcr.io/hosts.toml
```toml
server = "https://gcr.io"
[host."https://harbor.example.com:8443/v2/gcr.io"]
capabilities = ["pull", "resolve"]
override_path = true
```
2. use `nerdctl login` to harbor.example.com:8443, optionally setup mitmproxy to view the api traffic
3. directly pull the image from harbor is successfully
```console
❯ sudo HTTPS_PROXY=http://127.0.0.1:8080 nerdctl --debug-full pull harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest
DEBU[0000] verifying process skipped
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d" error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching image="harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest"
DEBU[0000] resolving host="harbor.example.com:8443"
DEBU[0000] do request host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest: resolving |--------------------------------------|
elapsed: 0.3 s total: 0.0 B (0.0 B/s)
DEBU[0000] fetch response received host="harbor.example.com:8443" response.header.content-length=152 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 14:05:20 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=7bc1d436bafba2a71e19ec32cfcc06b8; Path=/; HttpOnly" response.header.www-authenticate="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" response.header.x-envoy-upstream-service-time=7 response.header.x-request-id=cb37ff75-4c43-4d62-a565-2d767adbd4d8 response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
DEBU[0000] Unauthorized header="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" host="harbor.example.com:8443"
DEBU[0000] do request host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.inde
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest: resolving |--------------------------------------|
elapsed: 2.0 s total: 0.0 B (0.0 B/s)
DEBU[0002] fetch response received host="harbor.example.com:8443" response.header.content-length=1163 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Tue, 27 Feb 2024 14:05:22 GMT" response.header.docker-content-digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8\"" response.header.server=istio-envoy response.header.set-cookie="sid=1fbb1be72a6e2fd04a4962cfd6997843; Path=/; HttpOnly" response.header.x-envoy-upstream-service-time=1660 response.header.x-request-id=6dfd0fa9-f711-4512-a26f-ed627a26ab73 response.status="200 OK" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest"
DEBU[0002] resolved desc.digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" host="harbor.example.com:8443"
DEBU[0002] fetch digest="sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1163
DEBU[0002] fetch digest="sha256:68c29634fe49724f94ed34f18224316f776392f7a5a4014969ac5798a2ec96dc" mediatype=application/vnd.docker.container.image.v1+json size=4843
DEBU[0002] layer unpacked duration="221.474µs" layer="sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964"
DEBU[0002] layer unpacked duration="87.424µs" layer="sha256:2d4828968d6ca90e549ac054b8330c28081e02b31574f663fe8dae9c03275103"
DEBU[0002] layer unpacked duration="62.677µs" layer="sha256:b28a9b13dc6dccb175978a3e34f0ee5ec435e1328b9ded306724e490388585e4"
DEBU[0002] layer unpacked duration="92.493µs" layer="sha256:031224e6222c780a9783d9d287f6c25e37f16c800ba4bbb13e7f9c77036fb811"
harbor.example.com:8443/gcr.io/cadvisor/cadvisor:latest: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:ddadf3e2fd880deb4e0f3606d34a0d9da1165e3801116075d98a1901635dc9e8: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:68c29634fe49724f94ed34f18224316f776392f7a5a4014969ac5798a2ec96dc: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.2 s total: 0.0 B (0.0 B/s)
```
4. pulling the same image from registry mirrors fails as 401
```console
❯ sudo HTTPS_PROXY=127.0.0.1:8080 nerdctl --debug-full pull gcr.io/cadvisor/cadvisor:latest
DEBU[0000] verifying process skipped
DEBU[0000] Found hosts dir "/etc/containerd/certs.d"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d" error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching image="gcr.io/cadvisor/cadvisor:latest"
DEBU[0000] loading host directory dir=/etc/containerd/certs.d/gcr.io
DEBU[0000] resolving host="harbor.example.com:8443"
DEBU[0000] do request host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] fetch response received host="harbor.example.com:8443" response.header.content-length=152 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 03:27:43 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=63cc905e0a57374c5292b8273645d06c; Path=/; HttpOnly" response.header.www-authenticate="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" response.header.x-envoy-upstream-service-time=3 response.header.x-request-id=d0e29292-1e9e-4f15-ab65-51d37f1dd851 response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] Unauthorized header="Bearer realm=\"https://harbor.example.com:8443/service/token\",service=\"harbor-registry\",scope=\"repository:gcr.io/cadvisor/cadvisor:pull\"" host="harbor.example.com:8443"
DEBU[0000] do request host="harbor.example.com:8443" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] fetch response received host="harbor.example.com:8443" response.header.content-length=198 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 27 Feb 2024 03:27:43 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=istio-envoy response.header.set-cookie="sid=5ce41f88f2ba4e9246430cfa8e46312f; Path=/; HttpOnly" response.header.www-authenticate="Basic realm=\"harbor\"" response.header.x-envoy-upstream-service-time=1 response.header.x-request-id=0c1dc612-bbaa-4638-ba67-18f0a69006cf response.status="401 Unauthorized" url="https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io"
DEBU[0000] Unauthorized header="Basic realm=\"harbor\"" host="harbor.example.com:8443"
DEBU[0000] resolving host=gcr.io
DEBU[0000] do request host=gcr.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.11+unknown request.method=HEAD url="https://gcr.io/v2/cadvisor/cadvisor/manifests/latest"
gcr.io/cadvisor/cadvisor:latest: resolving |--------------------------------------|
elapsed: 133.1s total: 0.0 B (0.0 B/s)
INFO[0133] trying next host error="failed to do request: Head \"https://gcr.io/v2/cadvisor/cadvisor/manifests/latest\": Bad Gateway" host=gcr.io
FATA[0133] failed to resolve reference "gcr.io/cadvisor/cadvisor:latest": unexpected status from HEAD request to https://harbor.example.com:8443/v2/gcr.io/cadvisor/cadvisor/manifests/latest?ns=gcr.io: 401 Unauthorized
```
5. Browse through the mitmproxy api requests, I can see the request to`https://harbor.example.com:8443/service/token?scope=repository%3Acadvisor%2Fcadvisor%3Apull&scope=repository%3Agcr.io%2Fcadvisor%2Fcadvisor%3Apull&service=harbor-registry` lacks the `Authorization: Basic xxxx` header, hence results the final 401 error.
I also checked the mitmproxy api traffic using `ctr` and `crictl`, on successfully image pulling from registry mirrors, the above `Authorization: Basic xxxx` header is both present.
### Describe the results you received and expected
`nerdctl pull` from private registry mirror works, the credential of the mirror registry can be passed and reused from nerdctl login credential and hence adding the missing `Authorization: Basic xxxx` header.
### What version of nerdctl are you using?
Client:
Version: 1.7.2
OS/Arch: linux/amd64
Git commit: e32c4b023bf41e5c8325cfb893a53cefb5fc68ed
buildctl:
Version:
Server:
containerd:
Version: v1.7.13
GitCommit: 7c3aca7a610df76212171d200ca3811ff6096eb8.m
runc:
Version: 1.1.12
### Are you using a variant of nerdctl? (e.g., Rancher Desktop)
None
### Host information
Client:
Namespace: default
Debug Mode: false
Server:
Server Version: v1.7.13
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Log: fluentd journald json-file syslog
Storage: native overlayfs
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.7.6-arch1-1
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 125GiB
Name: arch
ID: 36b0778a-4ce8-4dca-af98-5afa21440138
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.