googleapis / googleapis/google-cloud-java

[java-bigquery] Bigquery: it should retry on BigQueryException: 403 Forbidden Exceeded rate limits

オープン
#12,169 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る
api: bigquery type: feature request
主要言語
Java
スター
2.1k
フォーク
1.2k
平均マージ
1日 23時間
マージ済み PR(30日)
154

説明

#### Environment details

1. OS type and version: linux (does not matter)
2. Java version: 1.8
3. version(s): 2.11.2 (But behaviour would be the same in newer versions afaiu, nos tested though)

#### Steps to reproduce

1. Make server return this response with status 403:

```json
{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"reason" : "rateLimitExceeded"
} ],
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"status" : "PERMISSION_DENIED"
}
```

2. Observe there are no retries

This have been tested mocking the server with wiremock and returning exactly that response, verifying the trace produced matched the production environment one

#### Code example

```java
// example
bigQueryService.listDatasets(
bigQueryConnection.getProjectId,
DatasetListOption.pageSize(Configuration.bigQueryDatasetPageSize)
)
```

#### Stack trace
```
Caused by: com.google.cloud.bigquery.BigQueryException: Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:299)
at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:778)
at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:775)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:774)
at .....
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://www.googleapis.com/bigquery/v2/projects/myproject/datasets/mydataset/tables/mytable?prettyPrint=false
{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"reason" : "rateLimitExceeded"
} ],
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"status" : "PERMISSION_DENIED"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:297)
... 44 common frames omitted
```

#### Any additional information below

I think the cause is the exception handling used in retry logic:

https://github.com/googleapis/java-bigquery/blob/f4e5fc59f4b9bc63c763ec1dc8b75f87defc9ced/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryBaseService.java#L29-L35

Afaiu that code makes abort retries in front of a `RuntimeException` and `BigQueryException` is so.

EDIT: Just i have observed the interceptor above could make BigqueryException retryable but 403 code is not in the list of retryable errors:

https://github.com/googleapis/java-bigquery/blob/f4e5fc59f4b9bc63c763ec1dc8b75f87defc9ced/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryException.java#L39-L41

I can see two alternatives if you think this should be fixed and the diagnostic is correct:
* open BigqueryOptions to allow users set a custom Exception handling
* Adds this case to the closed exception handler linked above

googleapis/java-bigquery#1498 asked to retry on a similiar error and was fixed but the return code was 400 in that case (instead 403)

Many thanks in advance

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。