mapbox / mapbox/magic-cfn-resources

Issues with spot fleet resource during update rollback

Open
#21 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
6
Forks
4
PR merge metrics
No merged PRs in 30d

Description

I encountered a scenario last night where a stack update attempted to change the fleet's Launch Specification. However there were syntax errors, and so the update to the custom resource failed. The result was that when the update rollback was complete, **the new fleet was considered obsolete and the old fleet request was canceled**. This led our system to terminate EC2s in the new spot fleet as soon as they launched.

The steps that led to this were complex, but after conversation with AWS support, the takeaways are:

1. When the custom resource's lambda function receives an UPDATE request, it **must** compare the incoming set of properties to the existing spot fleet. If they are identical, then it should not create a new spot fleet, and should not provide CloudFormation with a new spot fleet request ID.

2. `"Ref"` to custom cfn resources is actually not supported or documented. You're supposed to use `"Fn::GetAtt:"` for any and all information that you need to pull from the custom resource. The fact that we use a `"Ref"` to the spot fleet resource as a way to derive the existing fleet's request ID is "a loophole" and it can't be trusted.

cc @mapbox/platform

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the custom resource's Lambda UPDATE handler and trace how it compares incoming properties, creates a spot fleet, and returns the fleet request ID. Reproduce the rollback scenario with an invalid Launch Specification, then verify that unchanged properties preserve the existing request and that the rollback no longer cancels the active fleet.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, javascript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.