apache / apache/hudi

Streamer error table writes each error record twice when error table write unification is enabled

Open
#19,938 1 comment 0 reactions 0 assignees View on GitHub
area:deltastreamer priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Problem

With `hoodie.errortable.write.unification.enabled` on, the streamer's pre-commit flow commits the error table first (`ErrorTableCommitter.commit`) and only afterwards sums the error-table write statuses for the record counts (`SuccessfulRecordCounter.compute`). The write client's commit releases the write-status RDD it was handed (`releaseResources` unpersists what the write persisted), so the later aggregation re-evaluates the RDD's lineage. That lineage is the bulk insert itself, so the executors run the write a second time and produce a second base file under the instant that has already completed.

The error table then reads two rows for every error record, while the commit metadata reports the right count. Observed with a `BaseErrorTableWriter` backed by `SparkRDDWriteClient.bulkInsert` + `commit`: the log shows a second `HoodieCreateHandle` for the error table's partition right after "Result of error table commit ... is true", writing a new file id under the same instant time.

### Expected

The error-table write statuses are materialized once, before the error table commit releases them, and the counting step reads from that materialized set. One base file per error-table write, one row per error record.

### Environment

Hudi master (1.3.0-SNAPSHOT), Spark 3.5. Affects any deployment using the unified error table write path.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the pre-commit flow through ErrorTableCommitter.commit and SuccessfulRecordCounter.compute, then inspect BaseErrorTableWriter with SparkRDDWriteClient.bulkInsert and commit. Verify when the write-status RDD is materialized and released. Done means one base file and one row per error record, with the committed record count unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.