FileIOChannelFactory.match() traverses entire parent directory recursively
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
I was running a pipeline that reads a single file from my local home directory.
The pipeline got stuck, and upon taking a stack snapshot, I noticed that it was stuck in FileIOChannelFactory.match().
The code currently works by traversing the whole parent directory of the requested filepattern and checking which files match the filepattern. In my case, that means traversing everything in my home directory, which is *a lot* (and includes remotely mounted directories).
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java#L109
This is very wasteful and should be fixed.
Imported from Jira [BEAM-1309](https://issues.apache.org/jira/browse/BEAM-1309). Original Jira may contain additional context.
Reported by: jkff.
Contributor guide
Research direction
Start in sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java at match(), and inspect how a single-file pattern causes the parent directory to be traversed. Reproduce the behavior with a local home-directory file pattern; done means matching the requested file without recursively scanning the entire parent directory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100