[ECR] [bug]: Pull through cache redirects to upstream
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
We are using ecr with pull through cache rule as a "proxy" for dockerhub images.
Our eks nodes doesn't have access to public internet and fetch images from ecr instead
This was working well for a long time but suddenly our pods break with ErrImagePull error
We found this in logs:
```
Failed to pull image "123456789012.dkr.ecr.eu-west-1.amazonaws.com/mirror/docker.io/library/busybox:1.35.0":
rpc error: code = DeadlineExceeded desc = failed to pull and unpack image
"123456789012.dkr.ecr.eu-west-1.amazonaws.com/mirror/docker.io/library/busybox:1.35.0": failed to copy:
httpReadSeeker: failed open: failed to do request:
Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/1c/1c10e93757391bd459976f6382e0c0e03a0af3f239cd745424b06d3b4737f60e/data?verify=1709809284-GBrKCqH8amFBS2uwccqDUXDhl9E%!D(MISSING)":
dial tcp 104.16.102.207:443: i/o timeout
```
So, it's look like ecr redirects(?) to upstream registry
Contributor guide
Research direction
The issue names no repository files or tests. Start with the ECR pull through cache rule, the EKS node environment, and the recorded ErrImagePull log showing a redirect to production.cloudflare.docker.com; reproduce the pull without public internet access and determine whether the redirect is expected or indicates a cache failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, kubernetes
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100