aws / aws/aws-toolkit-azure-devops

CloudFormationCreateOrUpdateStack intermittently reports that the update failed even though the stack update is still in progress

Open
#423 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
258
Forks
114
PR merge metrics
No merged PRs in 30d

Description

`CloudFormationCreateOrUpdateStack` intermittently reports that the update failed even though the stack update is still in progress

**Screenshots**
Compare the timestamps of the Azure Pipelines logs and of the CloudFormation stack screenshots:
> 2021-07-16T00:04:25.1669564Z ##[section]Starting: Deploy CloudFormation template artifact
> 2021-07-16T00:04:25.1682103Z ==============================================================================
> 2021-07-16T00:04:25.1682446Z Task : AWS CloudFormation Create/Update Stack
> 2021-07-16T00:04:25.1682789Z Description : Creates a new AWS CloudFormation stack or updates the stack if it exists.
> 2021-07-16T00:04:25.1683056Z Version : 1.11.0
> 2021-07-16T00:04:25.1683264Z Author : Amazon Web Services
> 2021-07-16T00:04:25.1684784Z Help : Please refer to [AWS CloudFormation User Guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/) for more details on working with AWS CloudFormation.
>
> More information on this task can be found in the [task reference](https://docs.aws.amazon.com/vsts/latest/userguide/cloudformation-create-update.html).
>
> ####Task Permissions
> This task requires permissions to call the following AWS service APIs (depending on selected task options, not all APIs may be used):
> * cloudformation:CreateChangeSet
> * cloudformation:CreateStack
> * cloudformation:DeleteChangeSet
> * cloudformation:DescribeChangeSet
> * cloudformation:DescribeStacks
> * cloudformation:DescribeStackResources
> * cloudformation:ExecuteChangeSet
> * cloudformation:UpdateStack
>
> The task may also require permissions to upload your application template to the specified Amazon S3 bucket.
> 2021-07-16T00:04:25.1686419Z ==============================================================================
> 2021-07-16T00:04:25.7914108Z Configuring credentials for task
> 2021-07-16T00:04:25.7917842Z ...configuring AWS credentials from service endpoint '`service_endpoint_id`'
> 2021-07-16T00:04:25.7921389Z ...endpoint defines role-based credentials for role ***.
> 2021-07-16T00:04:25.7938858Z Configuring region for task
> 2021-07-16T00:04:25.7941522Z ...configured to use region us-east-2, defined in task.
> 2021-07-16T00:04:25.8059834Z Configuring credentials for task
> 2021-07-16T00:04:25.8105840Z ...configuring AWS credentials from service endpoint '`service_endpoint_id`'
> 2021-07-16T00:04:25.8106878Z ...endpoint defines role-based credentials for role ***.
> 2021-07-16T00:04:25.8107542Z Configuring region for task
> 2021-07-16T00:04:25.8108460Z ...configured to use region us-east-2, defined in task.
> 2021-07-16T00:04:25.8225072Z Checking existence for stack `stack_name`
> 2021-07-16T00:04:26.1316421Z Stack exists, switching to update stack mode
> 2021-07-16T00:04:26.1322549Z Updating stack with template file
> 2021-07-16T00:04:26.1909771Z Generated url to template: `url`
> 2021-07-16T00:04:26.1914834Z Loading template parameters from task definition
> 2021-07-16T00:04:26.1968920Z Successfully loaded template parameters
> 2021-07-16T00:04:26.1985316Z Setting capability CAPABILITY_AUTO_EXPAND for stack
> 2021-07-16T00:04:26.8707078Z Waiting for stack `stack_name` to reach update complete status
> 2021-07-16T00:08:28.5549623Z Stack update request failed with error: 'Stack `stack_name` failed to reach update completion status, error: 'Resource is not in the state stackUpdateComplete'' Error: Stack `stack_name` failed to reach update completion status, error: 'Resource is not in the state stackUpdateComplete'
> 2021-07-16T00:08:28.5604512Z ##[error]Error: Stack `stack_name` failed to reach update completion status, error: 'Resource is not in the state stackUpdateComplete'
> 2021-07-16T00:08:28.5620817Z at Object. (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.11.0/CloudFormationCreateOrUpdateStack.js:55:3093243)
> 2021-07-16T00:08:28.5621443Z at throw (native)
> 2021-07-16T00:08:28.5622282Z at n (/home/vsts/work/_tasks/CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99/1.11.0/CloudFormationCreateOrUpdateStack.js:1:781)
> 2021-07-16T00:08:28.5623677Z at process._tickDomainCallback (internal/process/next_tick.js:135:7)
> 2021-07-16T00:08:28.5628101Z ##[section]Finishing: Deploy CloudFormation template artifact

![image](https://user-images.githubusercontent.com/1730871/125874088-194397ec-9ff8-4551-8d3d-e0f322d4a9e3.png)
![image](https://user-images.githubusercontent.com/1730871/125874126-f62046b1-e715-4ba5-b7ee-2c6c35f720af.png)
![image](https://user-images.githubusercontent.com/1730871/125874178-058511d7-7e66-497d-89dc-b80506c9d5f4.png)

**Your Environment**

- On-prem or cloud based?: Cloud
- AWS Toolkit for Azure DevOps version: 1.11.0

Contributor guide

Open the contributing guide

Research direction

Start at the CloudFormationCreateOrUpdateStack entry point, using the bundled path shown in the log (CloudFormationCreateOrUpdateStack.js:55) to locate its source and waiting logic. Reproduce an update that remains in progress; done means the task does not report failure while CloudFormation is still updating and reports completion only after the final status.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.