Can't enforce 1 task at a time.
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
I'm trying to enforce a strict "one task at a time" for a particular ECS service. The task requires a lengthy shutdown process (1 minute to an hour) before it can safely exit. Having 0 tasks for a short time is perfectly fine. It is vital that, for this particular service, no more than 1 task run at a time, and that it follows the potentially lengthy shutdown procedure. No matter what I do, ECS is starting new tasks too early.
### Description
On the service, I have "minimum healthy percent" set to 0, and "maximum percent" set to 100. This should ensure 1 task at a time, right?
The entrypoint script traps the SIGTERM to start the shutdown process and wait for it to complete. As such, I have a very long "stop timeout" parameter in my container definition. This method is working correctly.
Unfortunately, ECS is deregistering the task as soon as the task-stop is sent, and then starting a new task. I can easily observe the old container still running after the ECS task is deregistered. While the container is still running, the task is not visible in the ECS console. This results in overlapping tasks.
### Expected Behavior
With "maximum percent" set to 100, I expect the task to be fully and completely gone before the next task is started. I expect all running containers to be visible in the ECS console. I expect the task to be deregistered after the container has actually exited.
### Observed Behavior
Tasks are deregistered when the `docker stop` command is sent, but before the container actually exits. In my case, this could be an hour later.
In addition, the ECS host with the unregistered running container is "stuck". It can't start new tasks until it exits. This is OK when the container is expected to be exiting shortly, but not OK if it's going to be an hour wait.
### Environment Details
Amazon ECS Agent - v1.40.0 (17e8d834)
Amazon Linux release 2 (Karoo)
ECS_ENABLE_CONTAINER_METADATA=true
ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=1h
ECS_CONTAINER_STOP_TIMEOUT=100m
ECS_ENABLE_UNTRACKED_IMAGE_CLEANUP=true
### Supporting Log Snippets
ECS-agent logs are perfectly normal. Tasks are transitioning their states appropriately. The problem is at the ECS service layer.
Contributor guide
Research direction
No repository files or tests are named. Start with the ECS service deployment behavior and the entrypoint script's SIGTERM handling, then review the stated minimum healthy percent, maximum percent, stop timeout, and cleanup settings. Done means ECS does not start an overlapping task, keeps the stopping task visible, and deregisters it only after the container exits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, shell
- Domain
- cloud, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100