bottlerocket-os / bottlerocket-os/bottlerocket

Bottlerocket ignoring settings.container-registry.mirrors

Open
#3,631 22 comments 19 reactions 0 assignees View on GitHub
status/research type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 18h
Merged PRs (30d)
12

Description

It seems like bottlerocket is ignoring `settings.container-registry.mirrors`
We have configured Pull through cache in aws ECR and set the `settings.container-registry.mirrors` for quay.io, docker.io, ghcr.io and registry.k8s.io to use ECR Pull through cache, but bottlerocket instance is still pulling directly from the upstream.
I can clearly see that mirrors were set using `apiclient -u /settings` on the instance.
It is not even trying to pull from ECR pull-through cache.
Our current setting is smth like this:
```
[[settings.container-registry.mirrors]]
"registry" = "registry.k8s.io"
"endpoint" = [".dkr.ecr.us-east-1.amazonaws.com/k8s"]
[[settings.container-registry.mirrors]]
"registry" = "quay.io"
"endpoint" = [".dkr.ecr.us-east-1.amazonaws.com/quay"]
[[settings.container-registry.mirrors]]
"registry" = "docker.io"
"endpoint" = [".dkr.ecr.us-east-1.amazonaws.com/dockerhub"]
[[settings.container-registry.mirrors]]
"registry" = "ghcr.io"
"endpoint" = [".dkr.ecr.us-east-1.amazonaws.com/github"]
```

**Image I'm using:**
bottlerocket-aws-k8s-1.27-x86_64-v1.16.1

**What I expected to happen:**
Pull through ECR cache for images in quay.io, docker.io, ghcr.io and registry.k8s.io registires.

**What actually happened:**
Pulling directly from upstream. Not even trying to pull from the ECR cache

**How to reproduce the problem:**
- Configure "Pull through cache" in ECR
- Set `settings.container-registry.mirrors` in your bottlerocket instance using UserData.
- Check that the setting is set by connecting to the instance either by ssh or Session Manager and run `apiclient -u /settings`
- Let your node in EKS pull any image from quay.io, docker.io, ghcr.io or registry.k8s.io registries.
- See containerd logs or k8s events for the deployed application

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the listed settings and verify them using `apiclient -u /settings`. Inspect containerd logs and Kubernetes events while pulling images from the named registries; done means the configured ECR pull-through cache is used instead of the upstream registry.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, linux
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.