Enable Consistent Data Push on Ingestion Jobs for REFRESH use case (Spark & Hadoop)
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Expand the feature in https://github.com/apache/pinot/issues/9268 from standalone mode only to Spark and Hadoop as well.
This entail extending `BaseSegmentPushJobRunner` for the following:
For Spark 2.4/3.2
`SparkSegmentMetadataPushJobRunner`
`SparkSegmentTarPushJobRunner`
`SparkSegmentUriPushJobRunner`
`HadoopSegmentMetadataPushJobRunner`
`HadoopSegmentTarPushJobRunner`
`HadoopSegmentUriPushJobRunner`
Adding postfix configuration in `HadoopSegmentGenerationJobRunner` as well as `SparkSegmentGenerationJobRunner`.
While also optionally adding integration tests coverage.
Contributor guide
Research direction
Start with issue #9268 and BaseSegmentPushJobRunner, then inspect the listed SparkSegmentMetadataPushJobRunner, SparkSegmentTarPushJobRunner, SparkSegmentUriPushJobRunner, and corresponding Hadoop runners. Review SparkSegmentGenerationJobRunner and HadoopSegmentGenerationJobRunner for postfix configuration. Done means the REFRESH data-push behavior works consistently in Spark 2.4/3.2 and Hadoop, with integration coverage if added.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100