apache / apache/hudi

Make write status idempotent

Open
#15,452 1 comment 0 reactions 0 assignees View on GitHub
area:index from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

HBase Index update some times not inconsistent with data. The main reason is that the result of task is not idempotent. A task run two times may get different bucket assign result. 
* Hudi on spark cache write status on executor. Once executor exits before commit, wrtie status will be regenerated. However, hbase index is updated by previous write status and will not be updated by new write status.
* When we use speculation in bulkinsert, hbase index is updated concurrently. Though only one task can succeed, it doesn't mean that all content in index is updated by this task. Those content updated by other failed task may be inconsistent with data.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4912
- Type: Bug

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how Spark tasks cache and regenerate write status, then follow the bulk-insert path where the HBase index is updated. Reproduce the executor-exit and speculation scenarios described in the issue. Done means repeated or concurrently attempted task execution leaves the HBase index consistent with the data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.