FileIO.read + deleteOnCommit functionality
Open
files
improvement
io
java
P3
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
A transforms that reads files, and deletes them once they've been consumed (and deals with retries appropriately).
FileIO.readMatches()
.withDeleteAfterReading()
Something like this
Imported from Jira [BEAM-10246](https://issues.apache.org/jira/browse/BEAM-10246). Original Jira may contain additional context.
Reported by: pabloem.
Contributor guide
Research direction
Start at the FileIO.readMatches() entry point and inspect its existing implementation and tests. Determine how a withDeleteAfterReading() option should behave when files are consumed and retried. Done means the transform supports deletion after reading while handling retries appropriately, with coverage for the stated behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100