awslabs / awslabs/data-on-eks

[feature] Use an ECR Pullthrough cache for Spark examples

Open
#823 0 comments 1 reaction 0 assignees View on GitHub
data-on-eks enhancement
Dominant language
Shell
Stars
857
Forks
303
Avg merge
11h 5m
Merged PRs (30d)
3

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

#### What is the outcome that you are trying to reach?

The Spark examples are pulling the same spark image but when running a large number of fresh instances I have seen some rate limiting and image pull issues. Using an ECR pull through cache seems like it would help reduce the issues while allowing the flexibility to use upstream images.

#### Describe the solution you would like

The spark operator blueprint could add an optional ECR pullthrough cache and rules to the vpc.tf file which then gives the details for the cache . We could then have commands in our docs to do the replacement of the images using like we do for the Bucket names.

#### Describe alternatives you have considered

#### Additional context

Contributor guide

Open the contributing guide

Research direction

Start by locating the Spark operator blueprint and its vpc.tf file, then review how the examples currently reference Spark images and how bucket-name replacements are documented. Define the optional ECR pull-through cache and rules there, and document the image replacement commands; done means fresh instances can use the cached images without losing upstream-image flexibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, spark, terraform
Domain
cloud, devops, documentation, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.