awslabs / awslabs/data-on-eks

Consolidate S3 usage in spark operator blueprint

Open
#868 0 comments 0 reactions 0 assignees View on GitHub
data-on-eks enhancement
Dominant language
Shell
Stars
857
Forks
303
Avg merge
11h 5m
Merged PRs (30d)
3

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

#### What is the outcome that you are trying to reach?

The Spark operator solution includes an S3 express bucket and a standard S3 bucket but the examples are split on where they are looking for source data and for the spark events. The spark history server is looking at the standard s3 bucket for events which isn't used in all the examples

#### Describe the solution you would like

Ideally all of the spark event logs would write to the same location and the S3 express bucket could be used for some examples to demonstrate the config/performance. Docs/code can be updated to make it clear which location is being used.

#### Describe alternatives you have considered

We've debated using solely one or the other bucket but having examples for both is handy.

#### Additional context

Contributor guide

Open the contributing guide

Research direction

Locate the Spark operator blueprint and its examples, then compare where source data and Spark event logs are configured; the issue names no files or tests, so begin by tracing those references. Done means the examples consistently document and configure their event-log location, while clearly showing where S3 Express is used.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, spark
Domain
cloud, data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.