CannotPullContainerError: pull image manifest has been retried 1 time(s) during routine retirement of your AWS Elastic Container Service tasks running on AWS Fargate
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
Routine retirement of your AWS Elastic Container Service tasks running on AWS Fargate is failing because ECS is unable to pull image due to invalid SHA digest for the respective image tag when creating tasks which were running before the AWS scheduled activity.
Steps to reproduce:
1. Deploy a service with ECR image before any scheduled AWS activities
2. AWS Scheduled activity creates a force update of the service with existing image and configuration and fails to find ECR image due to invalid SHA digest
Expectation:
AWS Scheduled activities should start ECS tasks in Service with same configuration and image as before the retirement of services for performing Fargate updates.
Actual:
CannotPullContainerError: pull image manifest has been retried 1 time(s): failed to resolve ref 12345678-eu-west-1.amazonaws.com/example-service:example-image:image-tag@sha256:sha digest: not found
Note: Image tag exists in ECR but has different SHA digest than the error message. We are using cross account ECR repo. Our work load account pulls image from deployment account ECR.
We found a workaround to fix this by creating a new image and pushing to ECR and re-deploying ECS Service with new image , however this is not a permanent fix when we have many parallel releases going on for our application.
Contributor guide
Research direction
Start by reviewing the reported ECS scheduled retirement flow and cross-account ECR image resolution, then reproduce the deployment and force-update steps described in the issue. Done means retired Fargate tasks start successfully with the existing service configuration and image rather than failing on an invalid SHA digest.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100