apache / apache/hudi

[SUPPORT] Apply 'write.ignore.failed' when write data failed

Open
#12,187 4 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

**Describe the problem you faced**

In Flink engine, if exception occurs when task writing data, it will be ignored and the exception will be reported to StreamWriteCoordinator with write event, StreamWriteCoordinator will decide whether to commit when there is write failure according to 'write.ignore.failed'.

Whether we should apply 'write.ignore.failed' ahead when write failure occurs, thus throw an exception faster.

**To Reproduce**

**Expected behavior**

1. promote 'write.ignore.failed' to hudi-client-common
2. throw exception in write handles
3. default value of 'write.ignore.failed' is false

**Environment Description**

**Additional context**

related PR:https://github.com/apache/hudi/pull/12150

**Stacktrace**

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Flink engine write handles and StreamWriteCoordinator described in the issue, then review the related PR #12150. Verify how write failures are reported and how `write.ignore.failed` is currently read; done means the setting is available in hudi-client-common, failures are handled there, and its default is false.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.