[feature] Use an ECR Pullthrough cache for Spark examples
- Dominant language
- Shell
- Stars
- 857
- Forks
- 303
- Avg merge
- 11h 5m
- Merged PRs (30d)
- 3
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
#### What is the outcome that you are trying to reach?
The Spark examples are pulling the same spark image but when running a large number of fresh instances I have seen some rate limiting and image pull issues. Using an ECR pull through cache seems like it would help reduce the issues while allowing the flexibility to use upstream images.
#### Describe the solution you would like
The spark operator blueprint could add an optional ECR pullthrough cache and rules to the vpc.tf file which then gives the details for the cache . We could then have commands in our docs to do the replacement of the images using like we do for the Bucket names.
#### Describe alternatives you have considered
#### Additional context
Contributor guide
Research direction
Start by locating the Spark operator blueprint and its vpc.tf file, then review how the examples currently reference Spark images and how bucket-name replacements are documented. Define the optional ECR pull-through cache and rules there, and document the image replacement commands; done means fresh instances can use the cached images without losing upstream-image flexibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, spark, terraform
- Domain
- cloud, devops, documentation, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100