(aws-batch): Internal failure when deploying compute environment changes
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
When invoking `cdk deploy` for a stack that includes Batch compute constructs, the deployment fails with the following message.
```
rjuengling-preloader: creating CloudFormation changeset...
11:15:36 PM | UPDATE_FAILED | AWS::Batch::JobQueue | rjuenglingpreloaderprocess0319A3D1
Internal Failure
❌ rjuengling-preloader-pipeline/rjuengling/preloader (rjuengling-preloader) failed: Error: The stack named rjuengling-preloader failed to deploy: UPDATE_ROLLBACK_COMPLETE: Internal Failure
at prepareAndExecuteChangeSet (/home/rjuengling/npm/node_modules/aws-cdk/lib/api/deploy-stack.ts:386:13)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at deployStack2 (/home/rjuengling/npm/node_modules/aws-cdk/lib/cdk-toolkit.ts:240:24)
at /home/rjuengling/npm/node_modules/aws-cdk/lib/deploy.ts:39:11
at run (/home/rjuengling/npm/node_modules/p-queue/dist/index.js:163:29)
```
### Expected Behavior
The deployment updates the stack without error.
### Current Behavior
The deployment fails with error `Internal Failure`, seemingly when CloudFormation is attempting to update a Batch job queue.
### Reproduction Steps
Create a stack with a Batch component and all stack resources tagged with some tag. Change the tag value, and run `cdk diff` to confirm that a stack update would entail a replacement of the job definition and job queue. Run `cdk deploy` to update the stack.
### Possible Solution
I believe the bug is not in CDK but in AWS CloudFormation. The failure occurs when the update is done by [replacement](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-replacement). This is the case when job definition's [tag](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-batch-jobdefinition.html#cfn-batch-jobdefinition-tags) or a job queue's [tag](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-batch-jobqueue.html#cfn-batch-jobqueue-tags) changes. Without changes in tags `cdk deploy` succeeds. This has been reported before [here](https://repost.aws/questions/QULDiQw0s-TwW2WKBcri6gTA/internal-error-on-cloud-formation-batch-job-definition-tags).
I examined the events initiated by CloudFormation in Cloud Trail. I see the expected event `CreateComputeEnvironment` which returns a new ARN but no event `CreateJobQueue`. So CloudFormation fails during the update of the job queue **before** creating the new job queue. I noticed `DescribeComputeEnvironments` events with the new ARN as input and a `null` response that coincide. It might be the case that the `null` response is unexpected and unhandled and is causing the update procedure to fail. I hope this helps with resolving the issue.
### Additional Information/Context
_No response_
### CDK CLI Version
2.41.0
### Framework Version
2.41.0
### Node.js Version
16.16.0
### OS
CentOS 7
### Language
Python
### Language Version
Python 3.8.11
### Other information
_No response_
Contributor guide
Research direction
Start with the reported reproduction: change a tag on a stack containing AWS Batch constructs, run `cdk diff`, then `cdk deploy` and inspect the CloudFormation and CloudTrail events described in the issue. Done would mean identifying a reproducible CDK-side change or confirming that the failure is external to CDK; the issue names no source file or test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100