Parititioned RLI take partition column value as a hint
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
h2. Functional requirement
in {{org.apache.hudi.metadata.HoodieBackedTableMetadata}} , we should provide 2 APIs for partitioned RLI lookup
class PartitionedRecordLevelIndexPrefixKey implements RawKey \{ private String recordKey; ... }
readPartitionedRecordIndexLocation( HoodieData recordKeys)
so it works for partitoned RLI. Behavior is for all record keys to lookup, it poke into all partitioned RLI file groups and do lookup there.
class PartitionedRecordLevelIndexKey implements RawKey \{ private String recordKey; private String partitionKey; ... } readPartitionedRecordIndexLocation( HoodieData recordKeys)
It takes pair of record key and partition key as input and only look into 1 file group that the key belongs to.
h3. Perf requirement
Also the lookup path should use the dynamic parallelism algorithm `org.apache.hudi.common.engine.HoodieEngineContext#mapGroupsByKey` implemented.
It should follow a similar flow of how global RLI lookup looks like.
No hacky implementation of collect large objects on driver/executor.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9648
- Type: Bug
- Fix version(s):
- 1.2.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in org.apache.hudi.metadata.HoodieBackedTableMetadata and trace the existing global RLI lookup flow. Read HoodieEngineContext#mapGroupsByKey and determine how both partitioned lookup inputs should route work without collecting large objects on the driver or executor. Done means both APIs support their stated lookup behavior and use dynamic parallelism consistently with the global flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100