apache / apache/beam

BigQuery IO Source is not Exporting to GCS as written in documentation

Open
#19,174 1 comment 0 reactions 0 assignees View on GitHub
bigquery bug core gcp P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Did some check on the beam code and find out that DataFlow is querying BigQuery and retrieve the result using pagination [1]. As per our understanding, this means no parallelism on reading BigQuery table. It is contradictory to what the documentation is telling us [2].
 
Is this some kind of work in progress? I'm filing as a bug since documentation telling me that it is using GCS meanwhile it's using NativeSourceReader which yield data per row as iterator.
 
[1] [https://github.com/apache/beam/blob/520b3a24e49306c30940ceab09100d775a04d28e/sdks/python/apache_beam/io/gcp/bigquery.py#L1083](https://github.com/apache/beam/blob/520b3a24e49306c30940ceab09100d775a04d28e/sdks/python/apache_beam/io/gcp/bigquery.py#L1083)
[2] [https://github.com/apache/beam/blob/520b3a24e49306c30940ceab09100d775a04d28e/sdks/python/apache_beam/io/gcp/bigquery.py#L60](https://github.com/apache/beam/blob/520b3a24e49306c30940ceab09100d775a04d28e/sdks/python/apache_beam/io/gcp/bigquery.py#L60)

Imported from Jira [BEAM-5352](https://issues.apache.org/jira/browse/BEAM-5352). Original Jira may contain additional context.
Reported by: rendybjunior.

Contributor guide

Open the contributing guide

Research direction

Start with sdks/python/apache_beam/io/gcp/bigquery.py at the NativeSourceReader reference near lines 60 and 1083, and compare the documented GCS behavior with the pagination behavior described in the issue. Determine whether the implementation or documentation is incorrect; done means the discrepancy is resolved and the documentation accurately describes the source behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.