BigQueryIO write is slow/fail with a bounded source
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
BigQueryIO Writer is slow / fail if the input source is bounded.
EDIT: Input BQ: 294 GB, 741,896,827 events
If the input source is bounded (GCS / BQ select / ...), BigQueryIO Writer use the "[Method.FILE_LOADS](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1168)" instead of streaming inserts.
Large amounts of input datas result in a java.lang.OutOfMemoryError / Java heap space (500 millions rows).
!PrepareWrite.BatchLoads.png|thumbnail!
We cannot use "Method.STREAMING_INSERTS" or control the batchs sizes since
[withMaxFilesPerBundle](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1131) is private :(
Someone reported a similar problem with GCS -\> BQ on Stackoverflow:
[Why is writing to BigQuery from a Dataflow/Beam pipeline slow?](https://stackoverflow.com/questions/45889992/why-is-writing-to-bigquery-from-a-dataflow-beam-pipeline-slow#comment78954153_45889992)
Imported from Jira [BEAM-2840](https://issues.apache.org/jira/browse/BEAM-2840). Original Jira may contain additional context.
Reported by: vspiewak.
Contributor guide
Research direction
Start in sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java, focusing on the bounded-source path that selects Method.FILE_LOADS and the withMaxFilesPerBundle setting. Reproduce the reported behavior with a large bounded GCS or BigQuery input and investigate the Java heap exhaustion. Done means bounded writes no longer fail or become excessively slow under the reported large-input conditions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100