Remove destination tables in BigQueryStorageQuerySource
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
In BigQueryStorageQuerySource, the results of a BigQuery query are written to a (named) temporary destination table rather than to an anonymous table since the BigQuery storage API can't currently read from all anonymous tables. When this API restriction is removed, then the BigQueryStorageQuerySource can start reading directly from anonymous result tables and most of the logic related to template compatibility and temporary dataset cleanup can be removed.
Imported from Jira [BEAM-6931](https://issues.apache.org/jira/browse/BEAM-6931). Original Jira may contain additional context.
Reported by: kjung520.
Contributor guide
Research direction
Start by locating BigQueryStorageQuerySource and review the original Jira BEAM-6931 for the missing context about the BigQuery Storage API restriction. Confirm that the restriction is removed, then identify the template-compatibility and temporary-dataset cleanup logic that can be removed; done means the source reads anonymous result tables without destination-table handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100