apache / apache/druid

Avoid file rename when pushing segments with HDFSDataPusher with UUID

Open
#7,532 2 comments 1 reaction 0 assignees View on GitHub
Area - Deep Storage Feature/Change Description Improvement
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description
As per the current behavior, HDFSDataSegmentPusher pushes segments to a temp location and then uses rename to move the segments to final location.
This behavior was added to fix race when two replica tasks try to write segment files to same location.

However when useUniquePath is set to true, a UUID is appended to the file name and the write to temp location is not required.

This change is to write directly to final location and rely on useUniquePath flag to avoid conflicts.

### Motivation

When underlying file system is S3AFileSystem, where rename is not supported a copy is done instead of rename and can be avoided when useUniquePath is set to true.
Link to S3A rename docs - https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1070

Contributor guide

Open the contributing guide

Research direction

Start by locating HDFSDataSegmentPusher and trace how useUniquePath controls temporary writes and the final rename. Compare the behavior with S3AFileSystem's rename limitations. Done means unique-path pushes write directly to the final location without the temporary copy or rename while preserving conflict avoidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, hadoop, java
Domain
backend, cloud, data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.