Improve error message in BigqueryIO when withNumFileShards is not specified
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
Reported in [https://lists.apache.org/thread.html/88c8da5eaf34ab1c7dc83372e0e2b1c31426b22aa4844b65b54ad16c@%3Cuser.beam.apache.org%3E](https://lists.apache.org/thread.html/88c8da5eaf34ab1c7dc83372e0e2b1c31426b22aa4844b65b54ad16c@%3Cuser.beam.apache.org%3E)
I am trying to use Method.FILE_LOADS for loading data into BQ in my streaming pipeline using 2.2.0. It looks like withNumFileShards needs to be also set for using this.
* Can we throw a better error message when it is not set? The exception thrown by the pipeline if this is not set is "Exception in thread "main" java.lang.IllegalArgumentException" without any message.
Imported from Jira [BEAM-3198](https://issues.apache.org/jira/browse/BEAM-3198). Original Jira may contain additional context.
Reported by: lcwik.
Contributor guide
Research direction
Start at BigqueryIO's Method.FILE_LOADS handling and trace the validation path used when withNumFileShards is absent. Reproduce the streaming pipeline configuration described in the issue, then verify that the resulting IllegalArgumentException includes a useful explanation of the missing option.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100