googleapis / googleapis/google-cloud-java

[java-bigquery] Bigquery: it should retry on BigQueryException: 403 Forbidden Exceeded rate limits

Đang mở
#12,169 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
api: bigquery type: feature request
Ngôn ngữ chính
Java
Star
2.1k
Fork
1.2k
Merge trung bình
1 ngày 23 giờ
Pull request đã merge (30 ngày)
157

Mô tả

#### Environment details

1. OS type and version: linux (does not matter)
2. Java version: 1.8
3. version(s): 2.11.2 (But behaviour would be the same in newer versions afaiu, nos tested though)

#### Steps to reproduce

1. Make server return this response with status 403:

```json
{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"reason" : "rateLimitExceeded"
} ],
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"status" : "PERMISSION_DENIED"
}
```

2. Observe there are no retries

This have been tested mocking the server with wiremock and returning exactly that response, verifying the trace produced matched the production environment one

#### Code example

```java
// example
bigQueryService.listDatasets(
bigQueryConnection.getProjectId,
DatasetListOption.pageSize(Configuration.bigQueryDatasetPageSize)
)
```

#### Stack trace
```
Caused by: com.google.cloud.bigquery.BigQueryException: Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:299)
at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:778)
at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:775)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:774)
at .....
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://www.googleapis.com/bigquery/v2/projects/myproject/datasets/mydataset/tables/mytable?prettyPrint=false
{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"reason" : "rateLimitExceeded"
} ],
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"status" : "PERMISSION_DENIED"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:297)
... 44 common frames omitted
```

#### Any additional information below

I think the cause is the exception handling used in retry logic:

https://github.com/googleapis/java-bigquery/blob/f4e5fc59f4b9bc63c763ec1dc8b75f87defc9ced/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryBaseService.java#L29-L35

Afaiu that code makes abort retries in front of a `RuntimeException` and `BigQueryException` is so.

EDIT: Just i have observed the interceptor above could make BigqueryException retryable but 403 code is not in the list of retryable errors:

https://github.com/googleapis/java-bigquery/blob/f4e5fc59f4b9bc63c763ec1dc8b75f87defc9ced/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryException.java#L39-L41

I can see two alternatives if you think this should be fixed and the diagnostic is correct:
* open BigqueryOptions to allow users set a custom Exception handling
* Adds this case to the closed exception handler linked above

googleapis/java-bigquery#1498 asked to retry on a similiar error and was fixed but the return code was 400 in that case (instead 403)

Many thanks in advance

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.