[EMR on EKS] [Feature Request]: Provide a minimal or base Spark image to reduce container size
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Please provide a **minimal or base version** of the Amazon EMR on EKS Spark Docker image.
The current Spark image (`public.ecr.aws/emr-on-eks/spark/emr-7.10.0:20250801-x86_64`) is approximately **2.2 GB** in size.
When building a custom application image on top of it, the total size increases to **~3.2 GB**, which significantly impacts **pod spin-up times** and **scaling performance** in Kubernetes environments.
A minimal or base image (with only core Spark runtime components) would help customers build leaner custom images and improve overall deployment efficiency.
**Which service(s) is this request for?**
EMR on EKS (PySpark)
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We use EMR on EKS for Spark-based workloads and build a custom image on top of the AWS-provided Spark image to include application logic and dependencies.
The resulting image size (~3.2 GB total) leads to:
- Slow pod startup times, especially during cluster autoscaling events
- High network bandwidth usage when pulling images
- Reduced elasticity and responsiveness of the cluster
Currently, there is no smaller base image available for EMR on EKS, making it difficult to optimize for speed and resource efficiency.
**Are you currently working around this issue?**
We are mitigating the problem by:
At the moment, there isn't a direct workaround for reducing the image size itself.
We host our custom image in Amazon ECR within the same region to minimize image pull latency.
AWS Support also suggested general Kubernetes optimizations such as image pre-pulling and using the IfNotPresent image pull policy, but these approaches only help marginally and do not address the core issue of the large base image size.
**Additional context**
- AWS Support confirmed that no minimal image is currently available.
- Reference EMR image: `public.ecr.aws/emr-on-eks/spark/emr-7.10.0:20250801-x86_64`
- Region: `us-east-1`
- EMR version: `7.10.0`
A lightweight or modular image option (for example, “base” vs “full”) would improve startup times, reduce image pull delays, and align EMR on EKS with best practices for containerized workloads.
**Attachments**
N/A
Contributor guide
Research direction
No repository files, tests, or entry points are identified in the issue. Start by reviewing the referenced EMR on EKS Spark image and the stated image-size constraints; done would be a maintained minimal or base image option with documented core contents and reduced size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, python, spark
- Domain
- cloud, data-engineering, devops, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100