[Batch] [Fargate] [request]: Allow override of `ephemeralStorage` in Batch Submit Job API
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Provide the option to override the `ContainerProperties.EphemeralStorage` setting in AWS Batch job definitions for Fargate when submitting individual jobs via the Batch SubmitJob API.
**Which service(s) is this request for?**
AWS Batch on AWS Fargate
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
The SubmitJob API currently supports overriding the vcpu and memory settings in the job definition when submitting individual jobs. Configurable ephemeral storage for AWS Batch fargate job definitions was released in Mar 2023, but the storage value cannot currently be overridden at job submit time.
My team maintains a batch processing application. Our jobs often use a common container and processing workflow, but resource requirements can differ greatly depending on the size of the input data and the processing options selected for a particular job.
The flexibility to override resource requirements on a job-by-job basis helps us cost-optimize our application without needing to create multiple job definitions. It also simplifies determining the resource requirements of newly developed workflows by quickly submitting the same job with a variety of available resources.
**Are you currently working around this issue?**
Current solution is to create a separate job definition for each desired `ephemeralStorage` value.
**Additional context**
Configurable ephemeral storage for AWS Batch fargate job definitions was released in Mar 2023:
https://aws.amazon.com/about-aws/whats-new/2023/03/aws-batch-configurable-ephemeral-storage-fargate/
https://github.com/aws/containers-roadmap/issues/1383
There is currently no option to override ephemeral storage in the `containerOverrides` field of the SubmitJob API
https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html
**Attachments**
Contributor guide
Research direction
Start with the AWS Batch SubmitJob API reference and the linked containers-roadmap issue 1383; compare the current containerOverrides options with the requested ephemeralStorage override. Done means AWS Batch on Fargate supports selecting ephemeral storage per submitted job without requiring separate job definitions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- api, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100