Short-circuit upsert operation when we know that the table is empty
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, tagging will be tagging every record irrespective of whether we're inserting into an empty table or not.
We obviously can short-circuit that and instead skip all of these operations speculatively provided that there's no updates possible and instead essentially pursue a bulk-insert pathway.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5500
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-3249
- Affects version(s):
- 0.12.2
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, entry points, or tests are identified. Trace the upsert tagging and bulk-insert paths, then determine how an empty table and the absence of possible updates can be established safely. Done means empty-table upserts skip per-record tagging and follow the bulk-insert path without changing behavior when updates are possible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100