[SUPPORT] Dirty data filtering failed
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
HI, I encountered an exception
I want to filter out this dirty data when entering the lake ，I tried 2 parameters, neither worked.
write.ignore.failed：true
hoodie.datasource.write.streaming.ignore.failed.batch : true
flink 1.13.5 hudi 0.13/0.10
How should we skip dirty data in this scenario?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported exception with Flink 1.13.5 and Hudi 0.13 or 0.10, using each of the two listed filtering parameters. Inspect the exception shown in the issue image and verify whether dirty records are skipped without failing ingestion; the issue is done when the supported configuration and behavior are documented or corrected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100