Push parallelism isn't implemented in Hadoop batch ingestion
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
In `HadoopSegmentTarPushJobRunner.run()`, this is the code at the very end of the method:
``` java
int pushParallelism = _spec.getPushJobSpec().getPushParallelism();
if (pushParallelism < 1) {
pushParallelism = segmentsToPush.size();
}
// Push from driver
try {
SegmentPushUtils.pushSegments(_spec, outputDirFS, segmentsToPush);
} catch (RetriableOperationException | AttemptsExceededException e) {
throw new RuntimeException(e);
}
```
So it doesn't actually use `pushParallelism`, and `SegmentPushUtils.pushSegments()` does a single-threaded (sequential) push.
Contributor guide
Research direction
Start in HadoopSegmentTarPushJobRunner.run() and trace how _spec.getPushJobSpec().getPushParallelism() is intended to affect SegmentPushUtils.pushSegments(). Verify the Hadoop batch ingestion path with different parallelism values; done means configured parallelism controls the push behavior rather than the current sequential call.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100