bottlerocket-os / bottlerocket-os/bottlerocket-core-kit
thar-be-registries: credentials entry creates a directory that shadows the `_default` wildcard mirror
- Dominant language
- Rust
- Stars
- 34
- Forks
- 77
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 23
Description
**Image I'm using:**
Bottlerocket 1.64.0, `aws-k8s-1.33` (containerd 2.2). Also present on `develop` at v16.3.0.
**What I expected to happen:**
With a wildcard mirror plus credentials for a private registry, pulls from that registry go to the mirror.
```toml
[[settings.container-registry.mirrors]]
registry = "*"
endpoint = ["http://127.0.0.1:30020"]
[[settings.container-registry.credentials]]
registry = "registry.example.com"
username = "user"
password = "pass"
```
**What actually happened:**
Pulls from `registry.example.com` bypass the mirror entirely and go upstream. Registries with no credentials entry use the mirror correctly. There is no warning, no log line, and the settings apply successfully, so this fails silently.
**How to reproduce the problem:**
Apply the settings above, then on the node:
```
ls /etc/containerd/certs.d/_default/ # hosts.toml
ls /etc/containerd/certs.d/registry.example.com/ # credentials.toml only, no hosts.toml
```
Pull an image from `registry.example.com` and observe no request reaching the mirror endpoint.
**Cause**
`write_hosts_toml` maps `registry = "*"` to `certs.d/_default/hosts.toml`. `write_credentials_toml` writes to `certs.d//credentials.toml` and calls `fs::create_dir_all` to do so, creating a directory that contains no `hosts.toml`.
containerd selects the host config directory by stat'ing the directory rather than by checking for `hosts.toml` (`core/remotes/docker/config/config_unix.go`, `HostDirFromRoot` in `hosts.go`):
```go
hosts = append(hosts,
filepath.Join(root, host),
filepath.Join(root, "_default"),
)
...
for _, p := range hostPaths(root, host) {
if _, err := os.Stat(p); err == nil {
return p, nil
}
}
```
The credentials-only directory matches first, so `_default` is never reached. `loadHostDir` then finds no `hosts.toml` there and returns no mirror hosts.
The effect is that a wildcard mirror silently stops applying to precisely the registries the user authenticates to. For anyone using a wildcard mirror for pull-through caching or P2P distribution, the private registries that benefit most are the ones excluded.
Docker Hub has an extra wrinkle: `write_credentials_toml` special-cases `docker.io` to `registry-1.docker.io`, which is also the host containerd resolves. A user-supplied `registry = "docker.io"` mirror writes to `certs.d/docker.io/`, which containerd never consults, so Docker Hub cannot be mirrored at all unless the user knows to write `registry-1.docker.io` in the mirror entry.
**Workaround**
Add an explicit mirror entry for every credentialed registry, duplicating the wildcard endpoints, and spell Docker Hub as `registry-1.docker.io`:
```toml
[[settings.container-registry.mirrors]]
registry = "registry.example.com"
endpoint = ["http://127.0.0.1:30020"]
```
`thar-be-registries` already supports both files in one directory, per `test_workflow_mirror_and_credential_same_registry`.
**Suggested fix**
Either of these would work, and I am happy to open a PR for whichever you prefer:
1. When a `*` mirror is configured, also write its `hosts.toml` into each credentialed registry's directory, with the appropriate `server` value for that registry rather than the `_default` form.
2. Have `write_credentials_toml` avoid creating a bare directory, for example by writing credentials into the `_default` directory keyed by host, if containerd's credential lookup allows it.
Option 1 is the smaller change and keeps the existing file layout.
At minimum, a warning at render time when a credentials entry has no matching mirror entry and a `*` mirror exists would turn a silent failure into a visible one.
Contributor guide
Research direction
Start with the write_hosts_toml and write_credentials_toml entry points, then read test_workflow_mirror_and_credential_same_registry. Reproduce the settings and inspect the generated certs.d directories, including the Docker Hub case. Done means credentialed registries still use the wildcard mirror without requiring duplicated mirror entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, rust
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100