aws / aws/containers-roadmap

[Batch] [Feature Request]: Avoid running duplicate jobs at the same time

Open
#2,283 0 comments 1 reaction 0 assignees View on GitHub
Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Avoid running jobs of the same kind at the same time and possibly run them sequentially.

**Which service(s) is this request for?**
Batch

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I have an architecture which executes batch jobs on fargate, these are scheduled jobs and are submitted to the batch queue by CloudWatch rules. So essentially, the architecture is CW -> Batch -> Fargate.

Let's say we have a CW rule C1 that submits jobs to a queue Q1 every 5 mins. C1 submits a jobs J1 at 1600 hrs to Q1. For some reason, this job is stuck in Q1 and is not able to be executed till 1605 hrs. At this points another job J2 is submitted to Q1 by C1. At 1606 hrs Q1 suddenly has capacity to execute both J1 and J2 and it does so. J1 and J2 are essential the same job meant to run at different times, but running at the same time they try to update the same database which could possibly causes data inconsistency issues.

So I want a way to tell batch that these are the same jobs (for example, setting some job parameter to identify same/similar jobs in queue) and so avoid executing them at the same time or just not executing the older instance of that job if new is available in the queue (i.e. execute J2 and ignore J1).

**Are you currently working around this issue?**
Allocating enough resources, having enough time gap in submitting jobs and database locking.

**Additional context**
N/A

**Attachments**
N/A

Contributor guide

Open the contributing guide

Research direction

The request concerns AWS Batch jobs submitted through CloudWatch rules to a Batch queue and executed on Fargate. Start by reviewing AWS Batch scheduling and job dependency or concurrency documentation, then identify whether the requested same-job behavior is supported. Done means documenting or delivering a defined way to prevent duplicate jobs from running concurrently or to prefer the newer queued job.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.