apache / apache/hudi

Improve API and configs for error Table

Open
#15,808 0 comments 0 reactions 0 assignees View on GitHub
from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

This is a follow up Jira to review comments in [https://github.com/apache/hudi/pull/7982/files]
The config names for error table can be generalised to accomadate for datasource and spark sql as well. Currently they are tied to DeltaStreamer. ([https://github.com/apache/hudi/pull/7982/files#r1118341097|https://github.com/apache/hudi/pull/7982/files#r1118341097)])
The API should throw exception so that any error cases can be handled. ([https://github.com/apache/hudi/pull/7982/files#r1118372238|https://github.com/apache/hudi/pull/7982/files#r1118372238)])
The constructor for error table writer can be generalised to handle DS and spark sql use cases as well. ([https://github.com/apache/hudi/pull/7982/files#r1118345162|https://github.com/apache/hudi/pull/7982/files#r1118345162)])

Add API for starting a commit. Handle concurrent error events or concurrent writes to the error table. ([https://github.com/apache/hudi/pull/7982/files#r1118350161|https://github.com/apache/hudi/pull/7982/files#r1118350161)])

Handle error records from writeStatusRDD ([https://github.com/apache/hudi/pull/7982/files#r1118380323])

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5858
- Type: Bug

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the linked PR 7982 and its comments, then trace the error table configuration, API, writer constructor, commit-start flow, and writeStatusRDD handling. The work is done when the API and configuration support datasource and Spark SQL use cases, concurrent error-table writes are handled, and error cases are surfaced consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
backend-api-design, data-engineering, stream-processing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.