Consolidate S3 usage in spark operator blueprint
- Dominant language
- Shell
- Stars
- 857
- Forks
- 303
- Avg merge
- 11h 5m
- Merged PRs (30d)
- 3
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
#### What is the outcome that you are trying to reach?
The Spark operator solution includes an S3 express bucket and a standard S3 bucket but the examples are split on where they are looking for source data and for the spark events. The spark history server is looking at the standard s3 bucket for events which isn't used in all the examples
#### Describe the solution you would like
Ideally all of the spark event logs would write to the same location and the S3 express bucket could be used for some examples to demonstrate the config/performance. Docs/code can be updated to make it clear which location is being used.
#### Describe alternatives you have considered
We've debated using solely one or the other bucket but having examples for both is handy.
#### Additional context
Contributor guide
Research direction
Locate the Spark operator blueprint and its examples, then compare where source data and Spark event logs are configured; the issue names no files or tests, so begin by tracing those references. Done means the examples consistently document and configure their event-log location, while clearly showing where S3 Express is used.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, spark
- Domain
- cloud, data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100