[ECS/Batch] [bug] Batch unmanaged compute env cant schedule jobs on a autoscaler enabled ECS cluster
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
We have the following setup:
1. AWS batch job with an unmanaged compute environment (we need this because we use GPU and custom disk sizes, and managed environment with Launch templates are not working. But that should be a different bug report).
2. The automatically created ECS cluster is enabled with a EC2 Autoscaling group Capacity provider.
3. The Autoscaling group has a target tracking dynamic scaling policy which tracks the metric `AWS/ECS/ManagedScaling > CapacityProviderReservation`.
The min size of the ASG is 0, so during idle periods there are no instances running on the ASG. When we schedule a Batch job at this point, the job becomes stuck in RUNNABLE state until we somehow manually trigger a scale up of the ASG. I think the reason is because Batch sees the ECS cluster and thinks well there are no resources to schedule the job, so I will keep the job in RUNNABLE state. But only if the Batch starts a Task on ECS, then ECS CapacityProviderReservation metric can change and it will trigger an scale up. One can see how it leads to a dead end. Now, we have verified all the related permissions and present and there are no capacity issues. If we scale up the ASG the job gets scheduled as expected. The scale down after the job finishes also works fine and the ASG/ECS cluster returns to 0 instances after.
Is this an expected behaviour? Is someone as well experiencing this? For now, we are triggering a scale up manually before we schedule the job. Will this Unmanaged Batch <-> ECS integration improve over time?
Thanks, and happy to provide more information if required.
Contributor guide
Research direction
Start by reproducing the AWS Batch unmanaged compute environment with an ECS cluster, an autoscaling group at minimum size 0, and the CapacityProviderReservation target-tracking policy. Observe whether a job remains RUNNABLE until the autoscaling group is manually increased. Done means the expected integration behavior is documented or an actionable service change is identified; the payload names no repository files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100