Fix HoodieIncr source checkpoint not progressing and add support from drop / cast columns
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
If there is no new commits to process in current batch , checkpoint value that is being returned is wrong .
Adding support to drop columns
Adding support to cast columns from [array , struct ] type to str
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3447
- Type: New Feature
---
## Comments
29/Mar/22 17:48;xushiyan;[~harsh1231] looks like there are 2 separate tasks in this ticket. can we split them?;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the HoodieIncr source checkpoint handling and the column-processing entry points. First separate the checkpoint fix from the column changes, then trace how empty batches and drop/cast requests are represented. Done means the checkpoint is correct when no commits are available and array or struct columns can be dropped or cast to strings, with tests covering each behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100