Bigquery queries should allow options to run in batch mode or not
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
When bigquery read does a query it sets the mode to batch. A batch query can be very slow to schedule as it batches it with other queries. However it doesn't use batch quota which is better for some cases. However, in some cases a fast query is better (especially in timed tests). It would be a good idea to have a configuration to the bigquery source to set this per-read.
Imported from Jira [BEAM-2817](https://issues.apache.org/jira/browse/BEAM-2817). Original Jira may contain additional context.
Reported by: laraschmidt.
Contributor guide
Research direction
Start by locating the BigQuery source read implementation and inspect how query mode is currently set. Review the linked BEAM-2817 Jira issue for additional context. Done means callers can configure batch versus non-batch execution per read, with behavior covered by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, sql
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100