Concurrent Image Pulls Thrash Disk, Causing `DockerTimeoutError: Could not transition to created; timed out after waiting`
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
I had an AWS Batch task fail:
```
Task failed to start - DockerTimeoutError: Could not transition to created; timed out after waiting 4m0s
```
I checked the host and ECS agent logs. One interesting host log from `journald`:
```
"INFO: task dockerd:8001 blocked for more than 122 seconds."
```
So dockerd was clearly hung, but why? Diving in further, I assembled a rough timeline of events:
- Many small jobs were submitted to an AWS Batch queue
- Batch spun up a `c6i.24xlarge` as an ECS instance to process those tasks
- As soon as the instance came online, it accepted 40 concurrent jobs, because it's a big instance
- ECS Agent kicked off 40 concurrent image pulls, each roughly 1-2gb, which saturated the root EBS volume
- I know this from logs like `"level=info time=2024-08-26T15:11:31Z msg=\"Pulling image for container concurrently\" container=\"default\" image=\"\" task=\"\""`
- Because the instance got overwhelmed, images couldn't be pulled, containers didn't transition to `created`, and tasks failed
- I can see from EBS metrics that the root volume got thrashed. I haven't dug into whether it was network-bound or disk-bound but based on some kernel logs I'm inclined to think disk. Anyway, it shouldn't matter.
I believe the problem is well understood but I'm curious if you all have guidance on how to remedy this.
I had a few ideas:
- Limit ECS concurrent pulls
- I imagine instead of pulling e.g. 40 images in parallel, it would be better to batch, e.g. 5, get the containers running, move on.
- I couldn't find an option to do this. And I wasn't sure about timeouts - if the container create timeout started before all the pulls started, there's no use batching, because you can't move enough bits to disk to get all 40 jobs started on time, batching or not. So you'd need to start the pull timeout from when the image starts pulling, and limit the number of containers in pulling state so that the others don't start their pull timers yet.
- Limit Docker concurrent pulls
- It looks like docker [does have the ability to limit concurrent downloads](https://stackoverflow.com/a/53915667)
- The idea here is that ECS would get bottlenecked on docker downloading a few images, starting the containers, and then ECS could move on to the next few images/containers
- Docker would be limiting within the ECS timeout window, i.e. by the time ECS was blocked, the clock would already be ticking and again there would be no way to start 40 tasks within that small window.
- Up the `ECS_CONTAINER_START_TIMEOUT`
- This would likely work, it would give docker enough time to pull all the images. But ideally concurrency would be limited so some tasks could get rolling while other images are downloading. Even better, ideally ECS would see back-pressure and schedule those tasks on other instances in the cluster.
- Up provisioned IOPS on EBS
- It's possible we could up IOPS to a high enough value to pull all the images in the allotted time window. That seems silly because this is really a bursty workload and sustained IOPS are not necessary.
- Set `GOMAXPROCS` to limit concurrent ECS agent operations
- This seems like a very heavy hammer - inhibiting all agent operations in case it tries to pull a bunch of images in parallel
- I'd be very nervous about what else might break if ECS agent was resource-starved
This is all to say - I believe I've found a scaling limit that cooks instances but no ratchet in ECS to avoid that limit. Curious if you all have suggestions to mitigate this scaling limit.
### Environment Details
ECS Agent: 1.85.3
Docker:
```
docker version
Client:
Version: 25.0.5
API version: 1.44
Go version: go1.22.5
Git commit: 5dc9bcc
Built: Thu Jul 25 00:00:00 2024
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 25.0.6
API version: 1.44 (minimum version 1.24)
Go version: go1.22.5
Git commit: b08a51f
Built: Thu Jul 25 00:00:00 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.11
GitCommit: 64b8a811b07ba6288238eefc14d898ee0b5b99ba
runc:
Version: 1.1.11
GitCommit: 4bccb38cc9cf198d52bebf2b3a90cd14e7af8c06
docker-init:
Version: 0.19.0
GitCommit: de40ad0
```
### Supporting Log Snippets
See above
Contributor guide
Research direction
No repository file or test is named; start by tracing ECS agent image-pull and container-start handling, then review the documented ECS_CONTAINER_START_TIMEOUT and Docker concurrent-download behavior. Done would require an agreed, tested mitigation for bursts of concurrent image pulls, such as configurable back-pressure or pull limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100