Avoid file rename when pushing segments with HDFSDataPusher with UUID
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Description
As per the current behavior, HDFSDataSegmentPusher pushes segments to a temp location and then uses rename to move the segments to final location.
This behavior was added to fix race when two replica tasks try to write segment files to same location.
However when useUniquePath is set to true, a UUID is appended to the file name and the write to temp location is not required.
This change is to write directly to final location and rely on useUniquePath flag to avoid conflicts.
### Motivation
When underlying file system is S3AFileSystem, where rename is not supported a copy is done instead of rename and can be avoided when useUniquePath is set to true.
Link to S3A rename docs - https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1070
Contributor guide
Research direction
Start by locating HDFSDataSegmentPusher and trace how useUniquePath controls temporary writes and the final rename. Compare the behavior with S3AFileSystem's rename limitations. Done means unique-path pushes write directly to the final location without the temporary copy or rename while preserving conflict avoidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, hadoop, java
- Domain
- backend, cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100