apache / apache/beam

Jobs that contain FileBasedSources with globs that expand into zero files should fail

Open
#18,168 0 comments 0 reactions 0 assignees View on GitHub
bug core java P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Currently we do not fail if a glob that is used to create a FileBasedSource expand into zero files. This can be error prone since a typo when specifying a glob pattern can result in a job that silently pass without producing any output. I believe this should be fixed as follows.

If a glob pattern specified as input to a FileBasedSource expand into zero files.
(1) If validation of FileBasedSource is enabled, fail the job at pipeline construction.
(2) If validation is disabled, fail the job at runtime by raising an exception at initial splitting.

Both Java and Python SDKs have to be fixed.

Imported from Jira [BEAM-1167](https://issues.apache.org/jira/browse/BEAM-1167). Original Jira may contain additional context.
Reported by: chamikara.

Contributor guide

Open the contributing guide

Research direction

Start at the Java and Python SDK implementations of FileBasedSource, tracing validation and the initial-splitting entry point. Verify behavior for a glob that expands to zero files: validation enabled should fail during pipeline construction, while disabled validation should raise at runtime during initial splitting; add or update coverage for both paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.