apache / apache/beam

FileIOChannelFactory.match() traverses entire parent directory recursively

Open
#18,193 0 comments 0 reactions 0 assignees View on GitHub
bug core java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I was running a pipeline that reads a single file from my local home directory.

The pipeline got stuck, and upon taking a stack snapshot, I noticed that it was stuck in FileIOChannelFactory.match().

The code currently works by traversing the whole parent directory of the requested filepattern and checking which files match the filepattern. In my case, that means traversing everything in my home directory, which is *a lot* (and includes remotely mounted directories).

https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java#L109

This is very wasteful and should be fixed.

Imported from Jira [BEAM-1309](https://issues.apache.org/jira/browse/BEAM-1309). Original Jira may contain additional context.
Reported by: jkff.

Contributor guide

Open the contributing guide

Research direction

Start in sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java at match(), and inspect how a single-file pattern causes the parent directory to be traversed. Reproduce the behavior with a local home-directory file pattern; done means matching the requested file without recursively scanning the entire parent directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.