"OutOfMemoryError: Direct buffer memory" in streaming dataflow at Google Cloud
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Hi ALL.
It seems there is some regression error in the last Beam version – 2.27.0.
There is a streaming dataflow that is runned in Google Cloud.
This dataflow is stable and has no any issues when built against Apache Beam version 2.23.0.
But if we build this dataflow against Apache Beam version 2.27.0 we catch an exception like below:
```
Error message from worker: java.lang.RuntimeException: java.lang.OutOfMemoryError: Direct buffer memory
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:932)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:972)
org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:195) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:149)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory java.base/java.nio.Bits.reserveMemory(Bits.java:175)
java.base/java.nio.DirectByteBuffer.(DirectByteBuffer.java:118) java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
org.conscrypt.ConscryptEngineSocket$SSLInputStream.(ConscryptEngineSocket.java:722) org.conscrypt.ConscryptEngineSocket.startHandshake(ConscryptEngineSocket.java:197)
org.conscrypt.ConscryptEngineSocket.waitForHandshake(ConscryptEngineSocket.java:547) org.conscrypt.ConscryptEngineSocket.getOutputStream(ConscryptEngineSocket.java:290)
java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:476) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
java.base/sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:265) java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367)
java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342) java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246)
com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:113) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:857)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1304)
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834)
```
The error is rare. 99% of a time the dataflow works without this (actually any) error.
This issue blocks us to make upgrade of the Beam library for the dataflow.
Imported from Jira [BEAM-11825](https://issues.apache.org/jira/browse/BEAM-11825). Original Jira may contain additional context.
Reported by: trunikov.
Contributor guide
Assessment
This issue has not been assessed yet.