apache / apache/hudi

Parititioned RLI take partition column value as a hint

Open
#17,116 1 comment 0 reactions 0 assignees View on GitHub
area:index from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

h2. Functional requirement
in {{org.apache.hudi.metadata.HoodieBackedTableMetadata}} , we should provide 2 APIs for partitioned RLI lookup
 

class PartitionedRecordLevelIndexPrefixKey implements RawKey \{ private String recordKey; ... }

readPartitionedRecordIndexLocation( HoodieData recordKeys)

 
so it works for partitoned RLI. Behavior is for all record keys to lookup, it poke into all partitioned RLI file groups and do lookup there.
 

class PartitionedRecordLevelIndexKey implements RawKey \{ private String recordKey; private String partitionKey; ... } readPartitionedRecordIndexLocation( HoodieData recordKeys)

 
It takes pair of record key and partition key as input and only look into 1 file group that the key belongs to.
 
h3. Perf requirement
Also the lookup path should use the dynamic parallelism algorithm `org.apache.hudi.common.engine.HoodieEngineContext#mapGroupsByKey` implemented.
 
It should follow a similar flow of how global RLI lookup looks like.
 
No hacky implementation of collect large objects on driver/executor.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9648
- Type: Bug
- Fix version(s):
- 1.2.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in org.apache.hudi.metadata.HoodieBackedTableMetadata and trace the existing global RLI lookup flow. Read HoodieEngineContext#mapGroupsByKey and determine how both partitioned lookup inputs should route work without collecting large objects on the driver or executor. Done means both APIs support their stated lookup behavior and use dynamic parallelism consistently with the global flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.