Partition update with global index in MOR tables resulting in duplicate values during read optimized queries
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
The way Hudi handles updating partition path is by locating the existing record and performing a delete on the previous partition and performing insert on new partition. In the case of Merge-on-Read tables the delete operation, and any update operation, is added as a log file. However since an insert occurs in the new partition the record is added in a parquet file. Querying using `QUERY_TYPE_READ_OPTIMIZED_OPT_VAL` fetches only parquet files and now we have the case where 2 records for given primary key are present
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1508
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Hudi-1508 JIRA report and trace how partition updates in Merge-on-Read tables handle the delete in the old partition and insert in the new partition. Reproduce the case with a global index and a read-optimized query, then verify that a primary key is returned only once after the fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100