Support atomic rename within FileSystem to replace inefficient Hadoop copy
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Hadoop copy operation is inefficient since it needs to stream the entirety of the resource through the machine performing the copy. Hadoop file system implementations do support an efficient rename.
Apache Beam sinks rely on being able to rename files atomically which is currently done by using FileSystem copy **** delete.
Imported from Jira [BEAM-2138](https://issues.apache.org/jira/browse/BEAM-2138). Original Jira may contain additional context.
Reported by: lcwik.
Contributor guide
Research direction
Locate the FileSystem abstraction and the sink path that currently performs rename through copy followed by delete. Read the Hadoop file system implementations to understand their rename support. Done means sinks can use an atomic rename without streaming the entire resource through the copying machine.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100