apache / apache/hudi

Partition update with global index in MOR tables resulting in duplicate values during read optimized queries

Open
#14,724 0 comments 0 reactions 0 assignees View on GitHub
area:index from-jira priority:blocker priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

The way Hudi handles updating partition path is by locating the existing record and performing a delete on the previous partition and performing insert on new partition. In the case of Merge-on-Read tables the delete operation, and any update operation, is added as a log file. However since an insert occurs in the new partition the record is added in a parquet file. Querying using `QUERY_TYPE_READ_OPTIMIZED_OPT_VAL` fetches only parquet files and now we have the case where 2 records for given primary key are present

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1508
- Type: Bug

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Hudi-1508 JIRA report and trace how partition updates in Merge-on-Read tables handle the delete in the old partition and insert in the new partition. Reproduce the case with a global index and a read-optimized query, then verify that a primary key is returned only once after the fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.