apache / apache/beam

BigQueryIO write is slow/fail with a bounded source

Open
#18,421 0 comments 0 reactions 0 assignees View on GitHub
bug gcp io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

BigQueryIO Writer is slow / fail if the input source is bounded.

EDIT: Input BQ: 294 GB, 741,896,827 events

If the input source is bounded (GCS / BQ select / ...), BigQueryIO Writer use the "[Method.FILE_LOADS](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1168)" instead of streaming inserts.

Large amounts of input datas result in a java.lang.OutOfMemoryError / Java heap space (500 millions rows).

!PrepareWrite.BatchLoads.png|thumbnail!

We cannot use "Method.STREAMING_INSERTS" or control the batchs sizes since
[withMaxFilesPerBundle](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1131) is private :(

Someone reported a similar problem with GCS -\> BQ on Stackoverflow:
[Why is writing to BigQuery from a Dataflow/Beam pipeline slow?](https://stackoverflow.com/questions/45889992/why-is-writing-to-bigquery-from-a-dataflow-beam-pipeline-slow#comment78954153_45889992)

Imported from Jira [BEAM-2840](https://issues.apache.org/jira/browse/BEAM-2840). Original Jira may contain additional context.
Reported by: vspiewak.

Contributor guide

Open the contributing guide

Research direction

Start in sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java, focusing on the bounded-source path that selects Method.FILE_LOADS and the withMaxFilesPerBundle setting. Reproduce the reported behavior with a large bounded GCS or BigQuery input and investigate the Java heap exhaustion. Done means bounded writes no longer fail or become excessively slow under the reported large-input conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.