[Batch] [Feature Request]: Avoid running duplicate jobs at the same time
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Avoid running jobs of the same kind at the same time and possibly run them sequentially.
**Which service(s) is this request for?**
Batch
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I have an architecture which executes batch jobs on fargate, these are scheduled jobs and are submitted to the batch queue by CloudWatch rules. So essentially, the architecture is CW -> Batch -> Fargate.
Let's say we have a CW rule C1 that submits jobs to a queue Q1 every 5 mins. C1 submits a jobs J1 at 1600 hrs to Q1. For some reason, this job is stuck in Q1 and is not able to be executed till 1605 hrs. At this points another job J2 is submitted to Q1 by C1. At 1606 hrs Q1 suddenly has capacity to execute both J1 and J2 and it does so. J1 and J2 are essential the same job meant to run at different times, but running at the same time they try to update the same database which could possibly causes data inconsistency issues.
So I want a way to tell batch that these are the same jobs (for example, setting some job parameter to identify same/similar jobs in queue) and so avoid executing them at the same time or just not executing the older instance of that job if new is available in the queue (i.e. execute J2 and ignore J1).
**Are you currently working around this issue?**
Allocating enough resources, having enough time gap in submitting jobs and database locking.
**Additional context**
N/A
**Attachments**
N/A
Contributor guide
Research direction
The request concerns AWS Batch jobs submitted through CloudWatch rules to a Batch queue and executed on Fargate. Start by reviewing AWS Batch scheduling and job dependency or concurrency documentation, then identify whether the requested same-job behavior is supported. Done means documenting or delivering a defined way to prevent duplicate jobs from running concurrently or to prefer the newer queued job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100