containerd / containerd/nerdctl

`nerdctl push` interop issue with Harbor registry

Open
#1,675 4 comments 5 reactions 0 assignees View on GitHub
area/login kind/external status/needs-design-discussion
Dominant language
Go
Stars
10.4k
Forks
826
Avg merge
1d 23h
Merged PRs (30d)
44

Description

### Description

Hi, I'm trying `nerdctl push` on a private Harbor registry with Bearer authentication and a custom token service, but the command fails with `401 Unauthorized`. After further investigation, it seems to be an interoperability issue between nerdctl and Harbor registries that have disabled basic authentication.

When nerdctl requests `GET /v2//blobs/`, the registry returns a `WWW-Authenticate` header that asks the client to request for a Bearer token with the scope of `repository::pull` from the specified `realm`. nerdctl would faithfully do so and retry the request with the right credentials, so far so good.

But when nerdctl reuses the same token for `POST /v2//blobs/uploads/` requests, the scope required is `repository::pull,push`, so this request is denied and the registry returns `401 Unauthorized`. The problem is that when a request contains a `Authorization` header but failed to authenticate, Harbor uses `Basic realm="harbor"` as the authentication challenge[[ref]](https://github.com/goharbor/harbor/blob/6113469a567623054a58d0f5506b90a55adca7ff/src/server/middleware/v2auth/auth.go#L95-L98), instead of the token service configured.

The registry I'm accessing exclusively relies on Bearer tokens for authentication and has been configured to deny all requests with basic auth. Being mislead by the new `WWW-Authenticate` header, all subsequent requests made by nerdctl would fail.

To solve this interop issue, nerdctl could do one of the following:

1. Make a "preflight" request to the `GET /v2/` API Version Check endpoint to retrieve a token without scope constraint and cache it for future requests.
2. Retry the request without an `Authorization` header. In this case, Harbor would return the correct `WWW-Authenticate` header that points to the token service.

On Harbor's side, there is https://github.com/goharbor/harbor/issues/17930, but no decision has been made yet.

### Steps to reproduce the issue

1. `nerdctl login ...`
2. `nerdctl push `

### Describe the results you received and expected

Received: `unexpected status from POST request to .../blobs/uploads/: 401 Unauthorized`

Expected: Command succeed without errors.

### What version of nerdctl are you using?

```
Client:
Version: v1.0.0
OS/Arch: linux/amd64
Git commit: c00780a1f5b905b09812722459c54936c9e070e6
buildctl:
Version: v0.10.5
GitCommit: bc26045116045516ff2427201abd299043eaf8f7

Server:
containerd:
Version: v1.6.8
GitCommit: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc:
Version: 1.1.4
GitCommit: 5fd4c4d144137e991c4acebb2146ab1483a97925
```

### Are you using a variant of nerdctl? (e.g., Rancher Desktop)

Rancher Desktop for macOS

### Host information

```
Client:
Namespace: default
Debug Mode: false

Server:
Server Version: v1.6.8
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Log: fluentd journald json-file syslog
Storage: native overlayfs
Security Options:
seccomp
Profile: default
Kernel Version: 5.15.78-0-virt
Operating System: Alpine Linux v3.16
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 5.809GiB
Name: lima-rancher-desktop
ID: 6238c1b1-ca3b-4e66-938c-d52f6c1895f5

WARNING: IPv4 forwarding is disabled
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.