apache / apache/hudi

To support partial update function which can move and update the data from the old partition to the new partition , when the data with same key change it's partition

Open
#14,862 0 comments 0 reactions 0 assignees View on GitHub
area:storage from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

to considerate such a scene, there 2 reocod *in different batch*  as follow 
||post_id ||position||weight||ts||day ||
| 1|shengzhen|3KG|1630480027|{color:#ff0000}20210901{color}|
| 1|beijing|3KG|1630652828|{color:#ff0000}20210903{color}|

 

when using the {color:#ff0000}*Global Index*{color} with such sql

 
{code:java}
merge into target_hudi_table  t
   using (
      select post_id, position, ts , day from source_table
   ) as s
on t.id = s.id
when natched then update set  t.position = s.position, t.ts=s.ts, t.day = s.day
when not matched then insert *
{code}
 

Beacuse now the hudi engine haven't support *cross partitions partial merge into,* the result in the target table is  

 
||post_id  (as primiary key)||position||weight||ts||day||
| 1|beijing| |1630652828|*{color:#ff0000}20210903{color}*|

the record still in  the old parition. 

 

but the *expected* result is 
||post_id  (as primiary key)||position||weight||ts||day||
| 1|beijing|*{color:#ff0000}3KG{color}*|1630652828|{color:#ff0000}*20210903*{color}|

 

 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-2441
- Type: Improvement

---

## Comments

21/Sep/21 12:05;David_Liang;we have do some job for this issue, The main development have finished,  It's testing now;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files or tests are named. Start by locating the global-index MERGE handling for partial updates across partitions and trace the two-record example; done means the updated record moves to the new partition while retaining the existing weight.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.