apache / apache/hudi

Move finalizeWrite outside the transaction lock during commit

Open
#19,715 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Context

Split out from the broader lock-scope discussion in #19587, to keep the design and testing focused.

### Problem

`commitStats`, `completeClustering`, `completeCompaction`, and `completeLogCompaction` perform several operations while completing a write or table service, including `resolveWriteConflict`, `finalizeWrite`, `writeTableMetadata`, and `saveAsComplete`.

`finalizeWrite` removes speculative or invalid files based on the markers for the current instant. When a write touches many partitions or files, `DirectWriteMarkers` may issue a large number of filesystem `list` operations, so `finalizeWrite` can hold the transaction lock for a long time when the filesystem is busy.

### Proposal

Move `finalizeWrite` outside the transaction boundary, keeping conflict resolution, Metadata Table updates, and timeline completion inside the lock. The sequence likes:

> finalize first → acquire the transaction lock → reload current table/timeline state → perform conflict resolution, Metadata Table update, and timeline completion in one critical section.

### Potential risk
It must **not** be split as `conflict-resolution-under-lock → unlock → finalize → relock`, because a commit completing in that gap invalidates the conflict check.

`finalizeWrite` is **not** read-only: marker reconciliation deletes invalid data files. Moving it outside the lock could race with rollback, cleaning, or another writer operating on the same files. We do not currently see an obvious correctness dependency requiring `finalizeWrite` itself to run under the lock, but this must be verified for all marker implementations and failure-recovery paths.

### Expected benefit

Reduce lock duration caused by filesystem operations during `finalizeWrite`, especially for large writes or when HDFS is under heavy load.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing commitStats, completeClustering, completeCompaction, and completeLogCompaction, focusing on finalizeWrite and the transaction-lock boundary. Review all marker implementations and failure-recovery paths to verify that finalizing before conflict resolution is safe. Done means reducing lock-held filesystem work without splitting conflict resolution from the protected critical section.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.