Move finalizeWrite outside the transaction lock during commit
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Context
Split out from the broader lock-scope discussion in #19587, to keep the design and testing focused.
### Problem
`commitStats`, `completeClustering`, `completeCompaction`, and `completeLogCompaction` perform several operations while completing a write or table service, including `resolveWriteConflict`, `finalizeWrite`, `writeTableMetadata`, and `saveAsComplete`.
`finalizeWrite` removes speculative or invalid files based on the markers for the current instant. When a write touches many partitions or files, `DirectWriteMarkers` may issue a large number of filesystem `list` operations, so `finalizeWrite` can hold the transaction lock for a long time when the filesystem is busy.
### Proposal
Move `finalizeWrite` outside the transaction boundary, keeping conflict resolution, Metadata Table updates, and timeline completion inside the lock. The sequence likes:
> finalize first → acquire the transaction lock → reload current table/timeline state → perform conflict resolution, Metadata Table update, and timeline completion in one critical section.
### Potential risk
It must **not** be split as `conflict-resolution-under-lock → unlock → finalize → relock`, because a commit completing in that gap invalidates the conflict check.
`finalizeWrite` is **not** read-only: marker reconciliation deletes invalid data files. Moving it outside the lock could race with rollback, cleaning, or another writer operating on the same files. We do not currently see an obvious correctness dependency requiring `finalizeWrite` itself to run under the lock, but this must be verified for all marker implementations and failure-recovery paths.
### Expected benefit
Reduce lock duration caused by filesystem operations during `finalizeWrite`, especially for large writes or when HDFS is under heavy load.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing commitStats, completeClustering, completeCompaction, and completeLogCompaction, focusing on finalizeWrite and the transaction-lock boundary. Review all marker implementations and failure-recovery paths to verify that finalizing before conflict resolution is safe. Done means reducing lock-held filesystem work without splitting conflict resolution from the protected critical section.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100