holdenk / holdenk/spark-flowchart
Documentation Idea: File Output Committer Algorithms
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 105
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Description
One possible idea for documentation or support for making jobs faster would be discussing various file output committers.
Particularly on object storage and as of Spark 3, using the default file output committer with s3a is going to result in a double data write and sad times.
Even registering s3a and friends via the right configs is something that I've had come up for a number of users, but that might be a little out of scope for the projects users.
I could try to contribute on the subject or just otherwise happy to throw it out there. Admittedly 9/10 on S3 if you just stick to the new cloud committers you're good. 😛
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing Spark application flowchart and the issue discussion about file output committers, object storage, Spark 3, and s3a configuration. No file or test is named; the documentation is done when it clearly covers the relevant committer algorithms, their object-storage behavior, and the applicable configuration scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, spark
- Domain
- cloud, data-engineering, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100