linkedin / linkedin/dynamometer
Blockgen job fails to clean up failed reduce attempts
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 134
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
The block generation job has custom output logic to allow each reducer to output to multiple block files.
When speculative execution is enabled, this can result in two copies of the same block file being generated (one of which may be incomplete). This can be worked around by setting mapreduce.reduce.speculative = false.
When a reducer attempt fails, the partial output files will not be cleaned up. I'm not aware of an easy workaround for this beyond manually cleaning up the files after the job completes.
We should have each reducer use a staging directory and only move the output files when it completes.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the block generation job's custom reducer output logic and how failed reducer attempts are handled. Verify that outputs are staged until successful completion and that failed attempts leave no partial block files, including when speculative execution is enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100