Add error handling logic flow for WriteStatus in spark sql writer
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
when commit contains error in WriteStatus in org.apache.hudi.HoodieSparkSqlWriter#commitAndPerformPostOperations, some logic flow can be added to handle error'ed records.
This can work jointly with error table HUDI-648. This can be generalized into error handling configuration: save error table, save to custom dead letter queue, fail and rollback, log and ignore.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5207
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at org.apache.hudi.HoodieSparkSqlWriter#commitAndPerformPostOperations and inspect how commit errors in WriteStatus are handled. Review the related error-table work in HUDI-648, then define and implement the desired configuration for saving an error table, sending records to a custom dead letter queue, failing and rolling back, or logging and ignoring them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100