Streamer error table writes each error record twice when error table write unification is enabled
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Problem
With `hoodie.errortable.write.unification.enabled` on, the streamer's pre-commit flow commits the error table first (`ErrorTableCommitter.commit`) and only afterwards sums the error-table write statuses for the record counts (`SuccessfulRecordCounter.compute`). The write client's commit releases the write-status RDD it was handed (`releaseResources` unpersists what the write persisted), so the later aggregation re-evaluates the RDD's lineage. That lineage is the bulk insert itself, so the executors run the write a second time and produce a second base file under the instant that has already completed.
The error table then reads two rows for every error record, while the commit metadata reports the right count. Observed with a `BaseErrorTableWriter` backed by `SparkRDDWriteClient.bulkInsert` + `commit`: the log shows a second `HoodieCreateHandle` for the error table's partition right after "Result of error table commit ... is true", writing a new file id under the same instant time.
### Expected
The error-table write statuses are materialized once, before the error table commit releases them, and the counting step reads from that materialized set. One base file per error-table write, one row per error record.
### Environment
Hudi master (1.3.0-SNAPSHOT), Spark 3.5. Affects any deployment using the unified error table write path.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the pre-commit flow through ErrorTableCommitter.commit and SuccessfulRecordCounter.compute, then inspect BaseErrorTableWriter with SparkRDDWriteClient.bulkInsert and commit. Verify when the write-status RDD is materialized and released. Done means one base file and one row per error record, with the committed record count unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100