False write success status on some errors
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
HoodieSparkSqlWriter [returns a success status|https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L337] in its {{write()}} method instead of throwing an exception when something fails (and such an error can legitimately happen, for example [here|https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L767]).
And there is a multitude of places where return values of this method are ignored (unit tests were omitted, but there is a lot of such cases over there as well):
[https://github.com/apache/hudi/blob/master/rfc/rfc-38/rfc-38.md#L77]
[https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala#L351]
[https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala#L148]
[https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala#L128]
[https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableDropPartitionCommand.scala#L62]
[https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/TruncateHoodieTableCommand.scala#L88]
This makes Hudi to swallow an error and report being successful on some write failures.
This issue actually happened in our infrastructure, but I don't have a good example to reproduce it, unfortunately.
For me it feels like a critical issue, please adjust the priority accordingly if I was not right :)
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5150
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala, inspecting write() near lines 337 and 767. Trace the listed callers, including MergeIntoHoodieTableCommand.scala, DefaultSource.scala, HoodieStreamingSink.scala, AlterHoodieTableDropPartitionCommand.scala, and TruncateHoodieTableCommand.scala, plus the existing unit tests. Done means write failures are not reported as successful by these paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100