googleapis / googleapis/google-cloud-java

[java-storage] HTTP: Read/socket timeouts are not being respected for some calls

Open
#12,628 0 comments 0 reactions 0 assignees View on GitHub
api: storage
Dominant language
Java
Stars
2.1k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
154

Description

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

- Search the issues already opened: https://github.com/googleapis/java-storage/issues
- Check for answers on StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform

If you are still having issues, please include as much information as possible:

#### Environment details

1. Specify the API at the beginning of the title. For example, "BigQuery: ...").
General, Core, and Other are also allowed as types
2. OS type and version: `Linux 6.8.0-1038-aws googleapis/java-storage#40-Ubuntu`
3. Java version: Java 11
4. version(s): 2.62.1

#### Steps to reproduce

Unclear but we find this happens when GCS buckets seem to be under more strain (slightly elevated 5XX rate)

#### Code example

```java
gs.retry.max-attempt: 0
gs.retry.total-timeout: 300000
gs.http.connect-timeout: 30000
gs.http.read-timeout: 30000
```

#### Stack trace
```
Caused by: com.google.cloud.storage.StorageException: Unknown Error
2026-02-26 18:26:52.359
at com.google.cloud.storage.UploadFailureScenario.toStorageException(UploadFailureScenario.java:128) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSessionPutTask.call(JsonResumableSessionPutTask.java:251) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
... 66 more
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$DefaultRetrier.run(Retrying.java:165) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$HttpRetrier.run(Retrying.java:206) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$RetrierWithAlgImpl.run(Retrying.java:137) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.put(JsonResumableSession.java:74) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.internalWrite(ApiaryUnbufferedWritableByteChannel.java:113) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.write(ApiaryUnbufferedWritableByteChannel.java:58) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.UnbufferedWritableByteChannelSession$UnbufferedWritableByteChannel.write(UnbufferedWritableByteChannelSession.java:39) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.DefaultBufferedWritableByteChannel.write(DefaultBufferedWritableByteChannel.java:121) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.StorageByteChannels$SynchronizedBufferedWritableByteChannel.write(StorageByteChannels.java:136) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:119) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
... 66 more
2026-02-26 18:26:52.359
Suppressed: com.google.cloud.storage.RetryContext$RetryBudgetExhaustedComment: Operation failed to complete within backoff budget (attempts: 1, elapsed: PT5M8.808S, nextBackoff: EXHAUSTED, timeout: PT5M)
2026-02-26 18:26:52.359
Caused by: java.net.SocketTimeoutException: Read timed out
2026-02-26 18:26:52.359
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.socketRead(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.lambda$put$0(JsonResumableSession.java:85) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.fill(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.read1(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.HttpURLConnection.getResponseCode(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpResponse.(NetHttpResponse.java:36) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSessionPutTask.call(JsonResumableSessionPutTask.java:122) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.lambda$put$0(JsonResumableSession.java:85) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$DefaultRetrier.run(Retrying.java:165) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$HttpRetrier.run(Retrying.java:206) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$RetrierWithAlgImpl.run(Retrying.java:137) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.put(JsonResumableSession.java:74) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.internalWrite(ApiaryUnbufferedWritableByteChannel.java:113) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.write(ApiaryUnbufferedWritableByteChannel.java:58) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.UnbufferedWritableByteChannelSession$UnbufferedWritableByteChannel.write(UnbufferedWritableByteChannelSession.java:39) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.DefaultBufferedWritableByteChannel.write(DefaultBufferedWritableByteChannel.java:121) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.StorageByteChannels$SynchronizedBufferedWritableByteChannel.write(StorageByteChannels.java:136) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:119) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) ~[?:?]
java.io.IOException: com.google.cloud.storage.StorageException: Unknown Error

2026-02-26 18:26:52.359
2026-02-26 18:26:52,354 WARN org.apache.flink.runtime.taskmanager.Task [] - EventTupleToEventRecordMapFunction -> SharedTransforms -> TimestampTransforms -> EmitMetricsFlatMapFunction -> TableIdsHandledMapFunction -> WatermarkProcessor -> FileSink: Writer -> FileSink: Committer (464/1536)#0 (2603fbe588f838f848f11d35b2177b35_20ba6b65f97481d5570070de90e4e791_463_0) switched from RUNNING to FAILED with failure cause:
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:122) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.storage.GSBlobStorageImpl$WriteChannel.write(GSBlobStorageImpl.java:291) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.writer.GSChecksumWriteChannel.write(GSChecksumWriteChannel.java:85) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.write(GSRecoverableFsDataOutputStream.java:162) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.DeflaterOutputStream.deflate(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.DeflaterOutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.GZIPOutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.hadoop.io.compress.GzipCodec$GzipOutputStream$ResetableGZIPOutputStream.write(GzipCodec.java:77) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.hadoop.io.compress.GzipCodec$GzipOutputStream.write(GzipCodec.java:118) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at java.io.OutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.formats.compress.writers.HadoopCompressionBulkWriter.addElement(HadoopCompressionBulkWriter.java:48) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.streaming.api.functions.sink.filesystem.BulkPartWriter.write(BulkPartWriter.java:51) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.connector.file.sink.writer.FileWriterBucket.write(FileWriterBucket.java:191) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.connector.file.sink.writer.FileWriter.write(FileWriter.java:198) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:166) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
```

#### External references such as API reference guides

- ?

#### Any additional information below

We get this error when the GCS bucket seems to be under strain and returning slightly elevated 5XX responses. It is punishing because it does not seem to respect our request/retry strategy. We're unsure what is going on, e.g. if the server is keeping the connection alive over the 5 minute span which causes us to not actually retry any more.

Following these steps guarantees the quickest resolution possible.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.