aws / aws/containers-roadmap

[ECR] [bug]: Pull through cache redirects to upstream

Open
#2,304 4 comments 8 reactions 0 assignees View on GitHub
ECR Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

We are using ecr with pull through cache rule as a "proxy" for dockerhub images.
Our eks nodes doesn't have access to public internet and fetch images from ecr instead
This was working well for a long time but suddenly our pods break with ErrImagePull error
We found this in logs:
```
Failed to pull image "123456789012.dkr.ecr.eu-west-1.amazonaws.com/mirror/docker.io/library/busybox:1.35.0":
rpc error: code = DeadlineExceeded desc = failed to pull and unpack image
"123456789012.dkr.ecr.eu-west-1.amazonaws.com/mirror/docker.io/library/busybox:1.35.0": failed to copy:
httpReadSeeker: failed open: failed to do request:
Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/1c/1c10e93757391bd459976f6382e0c0e03a0af3f239cd745424b06d3b4737f60e/data?verify=1709809284-GBrKCqH8amFBS2uwccqDUXDhl9E%!D(MISSING)":
dial tcp 104.16.102.207:443: i/o timeout
```
So, it's look like ecr redirects(?) to upstream registry

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files or tests. Start with the ECR pull through cache rule, the EKS node environment, and the recorded ErrImagePull log showing a redirect to production.cloudflare.docker.com; reproduce the pull without public internet access and determine whether the redirect is expected or indicates a cache failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, kubernetes
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.