aws / aws/containers-roadmap

[ECS/Fargate][bug]: Unable to start new tasks on platform 1.4.0 suddenly

Open
#1,425 1 comment 6 reactions 0 assignees View on GitHub
Fargate Fargate PV1.4 Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**

We have been running a few ECS services on fargate using platform 1.4.0 for the past ~3months now with no problem, expect for yesterday one of our ECS services has been unable to start any new tasks on platform 1.4.0. Can see the error below each tasks throughs when trying to reach `RUNNING` state.

```
(CannotCreateVolumeError: unable to copy contents to volume on host: containerd: failed to get image reference: context deadline exceeded)
```

> TO NOTE
The service in question does not makes use of any volume sharing between containers and none of our dockerfiles expose `VOLUME` paths either.

**Which service(s) is this request for?**
Fargate, ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I am trying to get our tasks to run on platform 1.4.0 again, but they keep on crashing when trying to start with the error as describe above. I have moved our platform version back to 1.3.0 for the service affected and that has allowed us to start the tasks again.

I updated the version again to 1.4.0, but I can watch the tasks just failing and trying to start again continuously. One task was able to start on 1.4.0 again, but none of the others could. So I am currently running a combination of 2 tasks on 1.3.0 and 1 on 1.4.0.

I have tested the same scenario as above on our staging aws account, that is a replica of our production environment, and I cannot replicate the error. I can switch between platform version as much as I like on our staging account with new tasks starting successfully each time. Which makes me believe this is either a bug with some sort of release recently, or our service is running on faulty infrastructure.

**Are you currently working around this issue?**
Yes, we are running our production service on platform 1.3.0

**Additional context**
Have added attachments of task definition and some screenshots of the current tasks running & stopped.

**Attachments**

[taskdefinition.txt](https://github.com/aws/containers-roadmap/files/6753762/taskdefinition.txt)
Screenshot 2021-07-01 at 13 54 17
![Screenshot 2021-07-01 at 13 54 47](https://user-images.githubusercontent.com/51319892/124257564-641f2380-db24-11eb-93a0-4983b5005d87.png)

Contributor guide

Open the contributing guide

Research direction

Review the attached taskdefinition.txt and task screenshots, then compare the failing production tasks on Fargate platform 1.4.0 with the working 1.3.0 and staging cases. Confirm the CannotCreateVolumeError and context deadline exceeded behavior; done means documenting a confirmed resolution or appropriate service escalation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.