aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap

[AWS::Batch::JobQueue] - [Enhancement] - Replacement Without Interrupting Running Jobs

Open
#2,463 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
No language data
Stars
1.1k
Forks
62
PR merge metrics
No merged PRs in 30d

Description

### Name of the resource

AWS::Batch::JobQueue

### Resource name

_No response_

### Description

### Problem

Updating properties that require replacement on `AWS::Batch::JobQueue` with a fixed `JobQueueName` fails with 409 ConflictException. CloudFormation's create-before-delete pattern tries to create a new queue with the same name before deleting the old one.

**Critical: Customers cannot safely update JobQueues without either deployment failures (409 errors) or interrupting running jobs (when using workarounds).**

Why customers use fixed names:
- External services submit jobs via AWS SDK using the queue name/ARN
- Stable identifiers required for third-party integrations

When customers attempt workarounds (like renaming), CloudFormation deletes the queue while jobs are still running, causing service disruption.

## Current Behavior

1. Update property requiring replacement (e.g., `Tags`)
2. CloudFormation attempts CreateJobQueue with same name
3. Fails with: `ConflictException: Object already exists (Status Code: 409)`
4. Stack rolls back to UPDATE_ROLLBACK_COMPLETE

## Properties Requiring Replacement

Per [CloudFormation docs](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-batch-jobqueue.html):
- `Tags` (most common)
- `JobQueueName`
- `JobQueueType`

Note: [Batch API](https://docs.aws.amazon.com/batch/latest/APIReference/API_TagResource.html) supports updating tags in-place via `TagResource`/`UntagResource`, but CloudFormation resource provider requires replacement.

## Requested Solution

**Primary: Graceful replacement that protects running jobs**

When replacement is required with a fixed name:
1. Disable old queue (prevents new submissions)
2. Wait for running jobs to complete (configurable timeout, default 30 min)
3. Delete old queue
4. Create new queue with same name
5. Enable new queue

If timeout reached: fail safely, leave queue DISABLED, allow customer to investigate.

**Alternative: Make `Tags` updateable in-place**

Use Batch `TagResource`/`UntagResource` APIs to avoid replacement entirely (eliminates most common trigger).

### Other Details

### Current Workarounds

**Documented workarounds:**
- Add `DeletionPolicy: Retain`, remove from stack, manually update tags, re-import

**Community ([GitHub #1229](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/1229#issuecomment-1407088842)):**
- Rename JobQueue (breaks external integrations)

**All workarounds:**
- Require manual intervention
- Break CloudFormation drift detection
- **Do not guarantee running jobs won't be interrupted**

## Related Issues

- [#1229](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/1229): JobQueue tag update failures (different error, same root cause)
- [#1655](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/1655): SchedulingPolicy tags require replacement
- [#1771](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/1771): Batch resources missing CloudFormation built-in tags

## Suggested Implementation

One possible approach:
- Detect fixed `JobQueueName` during replacement
- Use delete-before-create with job draining:
- `UpdateJobQueue` to DISABLED
- Poll `ListJobs` until all jobs reach terminal states
- Delete only after jobs complete
- Fail safely if timeout reached (leave DISABLED, don't delete)
- Consider in-place tag updates as quick win

## Reproduction Steps

**1. Create initial stack with fixed JobQueue name:**

```yaml
AWSTemplateFormatVersion: '2010-09-09'
Resources:
ComputeEnvironment:
Type: AWS::Batch::ComputeEnvironment
Properties:
Type: MANAGED
ComputeResources:
Type: FARGATE
MaxvCpus: 256
Subnets:
- !Ref Subnet
SecurityGroupIds:
- !Ref SecurityGroup
State: ENABLED

JobQueue:
Type: AWS::Batch::JobQueue
Properties:
JobQueueName: my-fixed-queue-name # Fixed name
Priority: 1
State: ENABLED
ComputeEnvironmentOrder:
- Order: 1
ComputeEnvironment: !Ref ComputeEnvironment
Tags:
Environment: dev

VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16

Subnet:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
CidrBlock: 10.0.1.0/24

SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Batch compute environment
VpcId: !Ref VPC
```

**2. Update the stack to trigger replacement (change tags):**

```yaml
# Change the Tags property:
Tags:
Environment: prod # Changed from dev
Owner: team-a # Added new tag
```

**3. Observe the error:**
- Stack update fails with `ConflictException: Object already exists (Status Code: 409)`
- CloudFormation attempts to create new queue with name `my-fixed-queue-name` before deleting old one

**4. To test with running jobs:**
- Submit jobs to the queue before step 2
- Observe that workarounds (like renaming) interrupt running jobs

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files, tests, or implementation entry points. Start with the AWS::Batch::JobQueue documentation and the referenced UpdateJobQueue, ListJobs, TagResource, and UntagResource APIs; done requires an agreed and validated way to replace or update a fixed-name queue without interrupting running jobs.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.