mapbox / mapbox/magic-cfn-resources
Issues with spot fleet resource during update rollback
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 6
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
I encountered a scenario last night where a stack update attempted to change the fleet's Launch Specification. However there were syntax errors, and so the update to the custom resource failed. The result was that when the update rollback was complete, **the new fleet was considered obsolete and the old fleet request was canceled**. This led our system to terminate EC2s in the new spot fleet as soon as they launched.
The steps that led to this were complex, but after conversation with AWS support, the takeaways are:
1. When the custom resource's lambda function receives an UPDATE request, it **must** compare the incoming set of properties to the existing spot fleet. If they are identical, then it should not create a new spot fleet, and should not provide CloudFormation with a new spot fleet request ID.
2. `"Ref"` to custom cfn resources is actually not supported or documented. You're supposed to use `"Fn::GetAtt:"` for any and all information that you need to pull from the custom resource. The fact that we use a `"Ref"` to the spot fleet resource as a way to derive the existing fleet's request ID is "a loophole" and it can't be trusted.
cc @mapbox/platform
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the custom resource's Lambda UPDATE handler and trace how it compares incoming properties, creates a spot fleet, and returns the fleet request ID. Reproduce the rollback scenario with an invalid Launch Specification, then verify that unchanged properties preserve the existing request and that the rollback no longer cancels the active fleet.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, javascript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100