aws / aws/aws-cdk

(aws-batch): Internal failure when deploying compute environment changes

Open
#21,988 6 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-batch bug documentation needs-cfn p3
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

When invoking `cdk deploy` for a stack that includes Batch compute constructs, the deployment fails with the following message.
```
rjuengling-preloader: creating CloudFormation changeset...
11:15:36 PM | UPDATE_FAILED | AWS::Batch::JobQueue | rjuenglingpreloaderprocess0319A3D1
Internal Failure

❌ rjuengling-preloader-pipeline/rjuengling/preloader (rjuengling-preloader) failed: Error: The stack named rjuengling-preloader failed to deploy: UPDATE_ROLLBACK_COMPLETE: Internal Failure
at prepareAndExecuteChangeSet (/home/rjuengling/npm/node_modules/aws-cdk/lib/api/deploy-stack.ts:386:13)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at deployStack2 (/home/rjuengling/npm/node_modules/aws-cdk/lib/cdk-toolkit.ts:240:24)
at /home/rjuengling/npm/node_modules/aws-cdk/lib/deploy.ts:39:11
at run (/home/rjuengling/npm/node_modules/p-queue/dist/index.js:163:29)
```

### Expected Behavior

The deployment updates the stack without error.

### Current Behavior

The deployment fails with error `Internal Failure`, seemingly when CloudFormation is attempting to update a Batch job queue.

### Reproduction Steps

Create a stack with a Batch component and all stack resources tagged with some tag. Change the tag value, and run `cdk diff` to confirm that a stack update would entail a replacement of the job definition and job queue. Run `cdk deploy` to update the stack.

### Possible Solution

I believe the bug is not in CDK but in AWS CloudFormation. The failure occurs when the update is done by [replacement](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-replacement). This is the case when job definition's [tag](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-batch-jobdefinition.html#cfn-batch-jobdefinition-tags) or a job queue's [tag](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-batch-jobqueue.html#cfn-batch-jobqueue-tags) changes. Without changes in tags `cdk deploy` succeeds. This has been reported before [here](https://repost.aws/questions/QULDiQw0s-TwW2WKBcri6gTA/internal-error-on-cloud-formation-batch-job-definition-tags).

I examined the events initiated by CloudFormation in Cloud Trail. I see the expected event `CreateComputeEnvironment` which returns a new ARN but no event `CreateJobQueue`. So CloudFormation fails during the update of the job queue **before** creating the new job queue. I noticed `DescribeComputeEnvironments` events with the new ARN as input and a `null` response that coincide. It might be the case that the `null` response is unexpected and unhandled and is causing the update procedure to fail. I hope this helps with resolving the issue.

### Additional Information/Context

_No response_

### CDK CLI Version

2.41.0

### Framework Version

2.41.0

### Node.js Version

16.16.0

### OS

CentOS 7

### Language

Python

### Language Version

Python 3.8.11

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the reported reproduction: change a tag on a stack containing AWS Batch constructs, run `cdk diff`, then `cdk deploy` and inspect the CloudFormation and CloudTrail events described in the issue. Done would mean identifying a reproducible CDK-side change or confirming that the failure is external to CDK; the issue names no source file or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.