apache / apache/hudi

Apply 'write.ignore.failed' when write data failed

Open
#16,690 1 comment 0 reactions 1 assignee Claimed by @cshuo View on GitHub
engine:flink priority:medium status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8400
- Type: Improvement
- Fix version(s):
- 1.2.0

---

## Comments

30/Oct/24 06:42;fhan688;{quote}We should clarify these items:
# should we promote the {{write.ignore.failed}} option to a common write config for each engine? Previously each eagine has it's own options and behavior.
# should we throw the exception in write handles or in the driver(after the write status are collected);
# should this option by default false or true?{quote}

# I agree. {{write.ignore.failed}} is a config in FlinkOptions and we promote it to HoodieWriteConfig in hudi-client-common module and named 'hoodie.write.ignore.failed' in this PR.
# I think fast fail is a better choice. In heavy PRD traffic job, several minutes late means huge amounts of records needs to be dealt with when restore.
# false, consider data quality.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.