apache / apache/hudi

Fix HoodieIncr source checkpoint not progressing and add support from drop / cast columns

Open
#15,028 1 comment 0 reactions 0 assignees View on GitHub
area:ingest from-jira priority:high type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

If there is no new commits to process in current batch , checkpoint value that is being returned is wrong .
Adding support to drop columns 

Adding support to cast columns  from [array , struct ] type to str

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3447
- Type: New Feature

---

## Comments

29/Mar/22 17:48;xushiyan;[~harsh1231] looks like there are 2 separate tasks in this ticket. can we split them?;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the HoodieIncr source checkpoint handling and the column-processing entry points. First separate the checkpoint fix from the column changes, then trace how empty batches and drop/cast requests are represented. Done means the checkpoint is correct when no commits are available and array or struct columns can be dropped or cast to strings, with tests covering each behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.