apache / apache/pinot

Push parallelism isn't implemented in Hadoop batch ingestion

Open
#6,505 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

In `HadoopSegmentTarPushJobRunner.run()`, this is the code at the very end of the method:

``` java
int pushParallelism = _spec.getPushJobSpec().getPushParallelism();
if (pushParallelism < 1) {
pushParallelism = segmentsToPush.size();
}
// Push from driver
try {
SegmentPushUtils.pushSegments(_spec, outputDirFS, segmentsToPush);
} catch (RetriableOperationException | AttemptsExceededException e) {
throw new RuntimeException(e);
}
```

So it doesn't actually use `pushParallelism`, and `SegmentPushUtils.pushSegments()` does a single-threaded (sequential) push.

Contributor guide

Open the contributing guide

Research direction

Start in HadoopSegmentTarPushJobRunner.run() and trace how _spec.getPushJobSpec().getPushParallelism() is intended to affect SegmentPushUtils.pushSegments(). Verify the Hadoop batch ingestion path with different parallelism values; done means configured parallelism controls the push behavior rather than the current sequential call.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.