[ECS] [request]: Fix `AssumeRole` failure race condition during `CreateCluster` ECS API operation
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
I’d like the ECS control plane to resolve or eliminate a longstanding, self‑imposed race condition related to creation and subsequent assumption of the `AWSServiceRoleForECS` service-linked role.
As requested in https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/818#issuecomment-1088110235, I am filing this issue here.
**Which service(s) is this request for?**
Amazon Elastic Container Service
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I would like the `CreateCluster` ECS API operation[^1] to wait for (or otherwise handle) the implicit creation of the `AWSServiceRoleForECS` service-linked role[^2], so that initial ECS cluster deployments for AWS accounts never fail with the `Unable to assume the service linked role` error (detailed below). The ECS API operation should guarantee that the role exists and is assumable by the time it attempts any setup work, introduce some retry logic, or increase the timeout before failure.
#### Current challenge
It is well-documented that the `AWSServiceRoleForECS` service-linked role for an AWS account is created by the ECS service when a cluster is first created[^2][^3]. It's therefore expected that this role does not exist when a customer attempts to create their first ECS cluster. When `CreateCluster` is called:
1. ECS calls the `CreateServiceLinkedRole` IAM API operation[^4].
2. IAM returns a `Role` response that acknowledges its creation, and begins replicating the change throughout AWS infrastructure[^4].
3. Asynchronously:
* IAM continues to propagate the new IAM role[^5].
* ECS calls the `AssumeRole` STS API operation[^6] using the returned service-linked role as the role to assume.
***--- The failure path frequently encountered by customers in this scenario: ---***
4. The IAM role change has not completed replication[^5].
5. The `AssumeRole` STS operation fails and returns an error[^7].
6. The ECS `CreateCluster` operation then ultimately fails, returning an `InvalidParameterException` error:
```plaintext
Unable to assume the service linked role. Please verify that the ECS service linked role exists.
```
The resulting hard failure is because of an unresolved and not easily understood or discoverable race condition. It isn’t triggered by user error or misconfiguration; any customer who creates an ECS cluster for the first time may run into this issue (and in my experience, many frequently do)[^8].
Because the `CreateServiceLinkedRole` operation success *is* eventually consistent[^5], and ECS does not roll back its creation after the subsequent `AssumeRole` failure, a retry usually succeeds. However, because these operations are not requested by the caller directly, the error is difficult to diagnose and is often assumed to be the result of some missing dependency or misconfiguration, resulting in false positive "fixes".
#### Why it matters
* **Widespread impact**: Any automated deployment CloudFormation, CLI, SDK that creates an ECS cluster can itermittently fail once per account, causing wasted time, customer confusion and dissatisfaction, and unnecessary retries[^8][^9].
* **Operational burden**: Deployers must attempt to understand what for them is likely an inscrutable error, or pre-create the role, adding complexity to CI/CD pipelines and documentation[^10].
* **Longstanding bug**: This race condition has been reported in multiple forums and persists across regions and SDK versions[^8][^9].
#### Impact of unresolved behavior
* **Slowed adoption**: New users evaluating ECS may be confused or frustrated by a spurious failure that is not easily diagnosed or preventable.
* **Increased support load**: Customers file support tickets or issues in template repositories to sidestep the race condition[^9].
* **Fragmented solutions**: Customers and partners re-implement their own workarounds or create superfluous dependencies rather than relying on a safe, built-in experience.
In short, please consider implementing some solution so that the service-linked role creation cannot cause a customer’s first attempt to fail. That change would eliminate a self-imposed race condition, simplify customer deployments, and lead to higher overall deployment success rates.
**Are you currently working around this issue?**
Yes. Our CloudFormation template documentation will warn of the potential error and instruct deployers to retry the stack creation.
**Additional context**
This race condition affects first-time ECS cluster creation for AWS customers using CloudFormation, AWS CLI, SDKs, and third-party tools (e.g. Terraform)[^8][^9].
The underlying IAM service-linked role API is non-idempotent[^4], so template creators are forced to document the potential error or implement custom retry or pre-creation logic.
Handling this within ECS’s control plane could permanently eliminate this self-imposed race condition, simplifying thousands of deployments.
---
**References**
[^1]: [CreateCluster API Reference – Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_CreateCluster.html)
[^2]: [Using Service-Linked Roles for Amazon ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-service-linked-roles.html)
[^3]: [AWS::ECS::Cluster – AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-cluster.html)
[^4]: [CreateServiceLinkedRole – AWS IAM API Reference](https://docs.aws.amazon.com/IAM/latest/APIReference/API_CreateServiceLinkedRole.html)
[^5]: [IAM eventual consistency](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-limits.html#reference_iam-limits-eventual-consistency)
[^6]: [AssumeRole – AWS STS API Reference](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html)
[^7]: [Common errors when using service-linked roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_roles.html#troubleshoot_roles-service-linked-errors)
[^8]: [GitHub Issue: ECS cluster creation fails with Unable to assume the service linked role](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/818)
[^9]: [GitHub Issue: ECS cluster creation fails intermittently](https://github.com/aws/containers-roadmap/issues/1246)
[^10]: [Terraform AWS provider issue: ECS cluster creation race condition](https://github.com/hashicorp/terraform-provider-aws/issues/8262)
Contributor guide
Assessment
This issue has not been assessed yet.