apache / apache/beam

Improve error message in BigqueryIO when withNumFileShards is not specified

Open
#18,516 0 comments 0 reactions 0 assignees View on GitHub
gcp improvement io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

Reported in [https://lists.apache.org/thread.html/88c8da5eaf34ab1c7dc83372e0e2b1c31426b22aa4844b65b54ad16c@%3Cuser.beam.apache.org%3E](https://lists.apache.org/thread.html/88c8da5eaf34ab1c7dc83372e0e2b1c31426b22aa4844b65b54ad16c@%3Cuser.beam.apache.org%3E)

I am trying to use Method.FILE_LOADS for loading data into BQ in my streaming pipeline using 2.2.0. It looks like withNumFileShards needs to be also set for using this.

* Can we throw a better error message when it is not set? The exception thrown by the pipeline if this is not set is "Exception in thread "main" java.lang.IllegalArgumentException" without any message.

Imported from Jira [BEAM-3198](https://issues.apache.org/jira/browse/BEAM-3198). Original Jira may contain additional context.
Reported by: lcwik.

Contributor guide

Open the contributing guide

Research direction

Start at BigqueryIO's Method.FILE_LOADS handling and trace the validation path used when withNumFileShards is absent. Reproduce the streaming pipeline configuration described in the issue, then verify that the resulting IllegalArgumentException includes a useful explanation of the missing option.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.