googleapis / googleapis/google-cloud-java

[java-storage] HTTP: Read/socket timeouts are not being respected for some calls

Abierto
#12,628 0 comentarios 0 reacciones 0 asignados Ver en GitHub
api: storage
Lenguaje dominante
Java
Estrellas
2.1k
Forks
1.2k
Merge medio
1 d 23 h
PR fusionados (30 d)
154

Descripción

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

- Search the issues already opened: https://github.com/googleapis/java-storage/issues
- Check for answers on StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform

If you are still having issues, please include as much information as possible:

#### Environment details

1. Specify the API at the beginning of the title. For example, "BigQuery: ...").
General, Core, and Other are also allowed as types
2. OS type and version: `Linux 6.8.0-1038-aws googleapis/java-storage#40-Ubuntu`
3. Java version: Java 11
4. version(s): 2.62.1

#### Steps to reproduce

Unclear but we find this happens when GCS buckets seem to be under more strain (slightly elevated 5XX rate)

#### Code example

```java
gs.retry.max-attempt: 0
gs.retry.total-timeout: 300000
gs.http.connect-timeout: 30000
gs.http.read-timeout: 30000
```

#### Stack trace
```
Caused by: com.google.cloud.storage.StorageException: Unknown Error
2026-02-26 18:26:52.359
at com.google.cloud.storage.UploadFailureScenario.toStorageException(UploadFailureScenario.java:128) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSessionPutTask.call(JsonResumableSessionPutTask.java:251) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
... 66 more
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$DefaultRetrier.run(Retrying.java:165) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$HttpRetrier.run(Retrying.java:206) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$RetrierWithAlgImpl.run(Retrying.java:137) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.put(JsonResumableSession.java:74) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.internalWrite(ApiaryUnbufferedWritableByteChannel.java:113) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.write(ApiaryUnbufferedWritableByteChannel.java:58) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.UnbufferedWritableByteChannelSession$UnbufferedWritableByteChannel.write(UnbufferedWritableByteChannelSession.java:39) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.DefaultBufferedWritableByteChannel.write(DefaultBufferedWritableByteChannel.java:121) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.StorageByteChannels$SynchronizedBufferedWritableByteChannel.write(StorageByteChannels.java:136) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:119) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
... 66 more
2026-02-26 18:26:52.359
Suppressed: com.google.cloud.storage.RetryContext$RetryBudgetExhaustedComment: Operation failed to complete within backoff budget (attempts: 1, elapsed: PT5M8.808S, nextBackoff: EXHAUSTED, timeout: PT5M)
2026-02-26 18:26:52.359
Caused by: java.net.SocketTimeoutException: Read timed out
2026-02-26 18:26:52.359
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.socketRead(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.lambda$put$0(JsonResumableSession.java:85) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.fill(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.read1(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.HttpURLConnection.getResponseCode(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpResponse.(NetHttpResponse.java:36) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSessionPutTask.call(JsonResumableSessionPutTask.java:122) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.lambda$put$0(JsonResumableSession.java:85) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$DefaultRetrier.run(Retrying.java:165) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$HttpRetrier.run(Retrying.java:206) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$RetrierWithAlgImpl.run(Retrying.java:137) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.put(JsonResumableSession.java:74) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.internalWrite(ApiaryUnbufferedWritableByteChannel.java:113) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.write(ApiaryUnbufferedWritableByteChannel.java:58) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.UnbufferedWritableByteChannelSession$UnbufferedWritableByteChannel.write(UnbufferedWritableByteChannelSession.java:39) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.DefaultBufferedWritableByteChannel.write(DefaultBufferedWritableByteChannel.java:121) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.StorageByteChannels$SynchronizedBufferedWritableByteChannel.write(StorageByteChannels.java:136) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:119) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) ~[?:?]
java.io.IOException: com.google.cloud.storage.StorageException: Unknown Error

2026-02-26 18:26:52.359
2026-02-26 18:26:52,354 WARN org.apache.flink.runtime.taskmanager.Task [] - EventTupleToEventRecordMapFunction -> SharedTransforms -> TimestampTransforms -> EmitMetricsFlatMapFunction -> TableIdsHandledMapFunction -> WatermarkProcessor -> FileSink: Writer -> FileSink: Committer (464/1536)#0 (2603fbe588f838f848f11d35b2177b35_20ba6b65f97481d5570070de90e4e791_463_0) switched from RUNNING to FAILED with failure cause:
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:122) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.storage.GSBlobStorageImpl$WriteChannel.write(GSBlobStorageImpl.java:291) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.writer.GSChecksumWriteChannel.write(GSChecksumWriteChannel.java:85) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.write(GSRecoverableFsDataOutputStream.java:162) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.DeflaterOutputStream.deflate(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.DeflaterOutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.GZIPOutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.hadoop.io.compress.GzipCodec$GzipOutputStream$ResetableGZIPOutputStream.write(GzipCodec.java:77) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.hadoop.io.compress.GzipCodec$GzipOutputStream.write(GzipCodec.java:118) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at java.io.OutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.formats.compress.writers.HadoopCompressionBulkWriter.addElement(HadoopCompressionBulkWriter.java:48) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.streaming.api.functions.sink.filesystem.BulkPartWriter.write(BulkPartWriter.java:51) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.connector.file.sink.writer.FileWriterBucket.write(FileWriterBucket.java:191) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.connector.file.sink.writer.FileWriter.write(FileWriter.java:198) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:166) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
```

#### External references such as API reference guides

- ?

#### Any additional information below

We get this error when the GCS bucket seems to be under strain and returning slightly elevated 5XX responses. It is punishing because it does not seem to respect our request/retry strategy. We're unsure what is going on, e.g. if the server is keeping the connection alive over the 5 minute span which causes us to not actually retry any more.

Following these steps guarantees the quickest resolution possible.

Thanks!

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.