apache / apache/hudi

Make user-defined BulkInsertPartitioner fit write path API

Open
#15,031 0 comments 0 reactions 0 assignees View on GitHub
area:writer from-jira priority:critical type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

this existing logic is problematic due to we can’t enforce user’s partitioner to return JavaRDD, this potentially breaks.

{code:java}
BulkInsertPartitioner partitioner = userDefinedBulkInsertPartitioner.isPresent()
? userDefinedBulkInsertPartitioner.get()
: BulkInsertInternalPartitionerFactory.get(config.getBulkInsertSortMode());
repartitionedRecords = (JavaRDD>) partitioner.repartitionRecords(dedupedRecords, parallelism);
{code}

The factory is used only in spark for now. So, we expect JavaRDD or HoodieData. The API can be made explicit about the constraint.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3463
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the write-path code shown around BulkInsertPartitioner, BulkInsertInternalPartitionerFactory, and the repartitionRecords call. Trace how JavaRDD and HoodieData are handled, then define the API constraint for user-defined partitioners. Done means the write path accepts only the supported return type without relying on the shown cast.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.