Jobs that contain FileBasedSources with globs that expand into zero files should fail
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Currently we do not fail if a glob that is used to create a FileBasedSource expand into zero files. This can be error prone since a typo when specifying a glob pattern can result in a job that silently pass without producing any output. I believe this should be fixed as follows.
If a glob pattern specified as input to a FileBasedSource expand into zero files.
(1) If validation of FileBasedSource is enabled, fail the job at pipeline construction.
(2) If validation is disabled, fail the job at runtime by raising an exception at initial splitting.
Both Java and Python SDKs have to be fixed.
Imported from Jira [BEAM-1167](https://issues.apache.org/jira/browse/BEAM-1167). Original Jira may contain additional context.
Reported by: chamikara.
Contributor guide
Research direction
Start at the Java and Python SDK implementations of FileBasedSource, tracing validation and the initial-splitting entry point. Verify behavior for a glob that expands to zero files: validation enabled should fail during pipeline construction, while disabled validation should raise at runtime during initial splitting; add or update coverage for both paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100