googleapis / googleapis/google-cloud-java

[java-storage] HTTP: Read/socket timeouts are not being respected for some calls

未关闭
#12,628 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
api: storage
主要语言
Java
星标
2.1k
派生
1.2k
平均合并
1 天 23 小时
30 天内合并 PR
154

描述

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

- Search the issues already opened: https://github.com/googleapis/java-storage/issues
- Check for answers on StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform

If you are still having issues, please include as much information as possible:

#### Environment details

1. Specify the API at the beginning of the title. For example, "BigQuery: ...").
General, Core, and Other are also allowed as types
2. OS type and version: `Linux 6.8.0-1038-aws googleapis/java-storage#40-Ubuntu`
3. Java version: Java 11
4. version(s): 2.62.1

#### Steps to reproduce

Unclear but we find this happens when GCS buckets seem to be under more strain (slightly elevated 5XX rate)

#### Code example

```java
gs.retry.max-attempt: 0
gs.retry.total-timeout: 300000
gs.http.connect-timeout: 30000
gs.http.read-timeout: 30000
```

#### Stack trace
```
Caused by: com.google.cloud.storage.StorageException: Unknown Error
2026-02-26 18:26:52.359
at com.google.cloud.storage.UploadFailureScenario.toStorageException(UploadFailureScenario.java:128) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSessionPutTask.call(JsonResumableSessionPutTask.java:251) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
... 66 more
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$DefaultRetrier.run(Retrying.java:165) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$HttpRetrier.run(Retrying.java:206) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$RetrierWithAlgImpl.run(Retrying.java:137) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.put(JsonResumableSession.java:74) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.internalWrite(ApiaryUnbufferedWritableByteChannel.java:113) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.write(ApiaryUnbufferedWritableByteChannel.java:58) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.UnbufferedWritableByteChannelSession$UnbufferedWritableByteChannel.write(UnbufferedWritableByteChannelSession.java:39) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.DefaultBufferedWritableByteChannel.write(DefaultBufferedWritableByteChannel.java:121) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.StorageByteChannels$SynchronizedBufferedWritableByteChannel.write(StorageByteChannels.java:136) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:119) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
... 66 more
2026-02-26 18:26:52.359
Suppressed: com.google.cloud.storage.RetryContext$RetryBudgetExhaustedComment: Operation failed to complete within backoff budget (attempts: 1, elapsed: PT5M8.808S, nextBackoff: EXHAUSTED, timeout: PT5M)
2026-02-26 18:26:52.359
Caused by: java.net.SocketTimeoutException: Read timed out
2026-02-26 18:26:52.359
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.socketRead(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.lambda$put$0(JsonResumableSession.java:85) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.fill(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.read1(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.HttpURLConnection.getResponseCode(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpResponse.(NetHttpResponse.java:36) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSessionPutTask.call(JsonResumableSessionPutTask.java:122) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.lambda$put$0(JsonResumableSession.java:85) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$DefaultRetrier.run(Retrying.java:165) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$HttpRetrier.run(Retrying.java:206) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$RetrierWithAlgImpl.run(Retrying.java:137) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.put(JsonResumableSession.java:74) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.internalWrite(ApiaryUnbufferedWritableByteChannel.java:113) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.write(ApiaryUnbufferedWritableByteChannel.java:58) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.UnbufferedWritableByteChannelSession$UnbufferedWritableByteChannel.write(UnbufferedWritableByteChannelSession.java:39) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.DefaultBufferedWritableByteChannel.write(DefaultBufferedWritableByteChannel.java:121) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.StorageByteChannels$SynchronizedBufferedWritableByteChannel.write(StorageByteChannels.java:136) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:119) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) ~[?:?]
java.io.IOException: com.google.cloud.storage.StorageException: Unknown Error

2026-02-26 18:26:52.359
2026-02-26 18:26:52,354 WARN org.apache.flink.runtime.taskmanager.Task [] - EventTupleToEventRecordMapFunction -> SharedTransforms -> TimestampTransforms -> EmitMetricsFlatMapFunction -> TableIdsHandledMapFunction -> WatermarkProcessor -> FileSink: Writer -> FileSink: Committer (464/1536)#0 (2603fbe588f838f848f11d35b2177b35_20ba6b65f97481d5570070de90e4e791_463_0) switched from RUNNING to FAILED with failure cause:
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:122) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.storage.GSBlobStorageImpl$WriteChannel.write(GSBlobStorageImpl.java:291) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.writer.GSChecksumWriteChannel.write(GSChecksumWriteChannel.java:85) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.write(GSRecoverableFsDataOutputStream.java:162) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.DeflaterOutputStream.deflate(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.DeflaterOutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.GZIPOutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.hadoop.io.compress.GzipCodec$GzipOutputStream$ResetableGZIPOutputStream.write(GzipCodec.java:77) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.hadoop.io.compress.GzipCodec$GzipOutputStream.write(GzipCodec.java:118) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at java.io.OutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.formats.compress.writers.HadoopCompressionBulkWriter.addElement(HadoopCompressionBulkWriter.java:48) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.streaming.api.functions.sink.filesystem.BulkPartWriter.write(BulkPartWriter.java:51) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.connector.file.sink.writer.FileWriterBucket.write(FileWriterBucket.java:191) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.connector.file.sink.writer.FileWriter.write(FileWriter.java:198) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:166) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
```

#### External references such as API reference guides

- ?

#### Any additional information below

We get this error when the GCS bucket seems to be under strain and returning slightly elevated 5XX responses. It is punishing because it does not seem to respect our request/retry strategy. We're unsure what is going on, e.g. if the server is keeping the connection alive over the 5 minute span which causes us to not actually retry any more.

Following these steps guarantees the quickest resolution possible.

Thanks!

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。