apache / apache/hudi

SQL Merge into to support Control updates

Open
#7,200 3 comments 0 reactions 0 assignees View on GitHub
area:sql priority:medium type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

We have use case where we get updates from source and we have to perform control updates by lookup in Target table.

1. For suppose if I have a column in Target table with valid value and If I receive update from source then we need to retain the value in target other wise update from source. This is like Coalesce(Target column,Source Column).
2. Similarly Other use case, If I receive valid value from Source then update it else retain value in target. This is like Coalesce(Source Column, Target Column)

Currently we are performing preprocess by Joining our Source Input Data with Target Hudi table and take care of those above control updates by retaining it either from Target or source. This preprocess stage which including additional join with heavy Target table is causing slowness or lot of delay. Also Spark is not intelligent to prune the partitions based on my Input Data.

Sample-Data-scenario

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Apache Hudi's SQL MERGE INTO handling and the source/target update flow described in the issue; no files, tests, or entry points are named. Define how target-versus-source COALESCE control updates should work without the current preprocessing join, then add coverage for both retention cases and verify the resulting merge behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.