Improve API and configs for error Table
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
This is a follow up Jira to review comments in [https://github.com/apache/hudi/pull/7982/files]
The config names for error table can be generalised to accomadate for datasource and spark sql as well. Currently they are tied to DeltaStreamer. ([https://github.com/apache/hudi/pull/7982/files#r1118341097|https://github.com/apache/hudi/pull/7982/files#r1118341097)])
The API should throw exception so that any error cases can be handled. ([https://github.com/apache/hudi/pull/7982/files#r1118372238|https://github.com/apache/hudi/pull/7982/files#r1118372238)])
The constructor for error table writer can be generalised to handle DS and spark sql use cases as well. ([https://github.com/apache/hudi/pull/7982/files#r1118345162|https://github.com/apache/hudi/pull/7982/files#r1118345162)])
Add API for starting a commit. Handle concurrent error events or concurrent writes to the error table. ([https://github.com/apache/hudi/pull/7982/files#r1118350161|https://github.com/apache/hudi/pull/7982/files#r1118350161)])
Handle error records from writeStatusRDD ([https://github.com/apache/hudi/pull/7982/files#r1118380323])
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5858
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the linked PR 7982 and its comments, then trace the error table configuration, API, writer constructor, commit-start flow, and writeStatusRDD handling. The work is done when the API and configuration support datasource and Spark SQL use cases, concurrent error-table writes are handled, and error cases are surfaced consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend-api-design, data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100