googleapis / googleapis/google-cloud-java

[java-bigquery] Bigquery: it should retry on BigQueryException: 403 Forbidden Exceeded rate limits

Aperta
#12,169 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
api: bigquery type: feature request
Lingua principale
Java
Stelle
2.1k
Fork
1.2k
Merge medio
1g 23h
PR unite (30g)
154

Descrizione

#### Environment details

1. OS type and version: linux (does not matter)
2. Java version: 1.8
3. version(s): 2.11.2 (But behaviour would be the same in newer versions afaiu, nos tested though)

#### Steps to reproduce

1. Make server return this response with status 403:

```json
{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"reason" : "rateLimitExceeded"
} ],
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"status" : "PERMISSION_DENIED"
}
```

2. Observe there are no retries

This have been tested mocking the server with wiremock and returning exactly that response, verifying the trace produced matched the production environment one

#### Code example

```java
// example
bigQueryService.listDatasets(
bigQueryConnection.getProjectId,
DatasetListOption.pageSize(Configuration.bigQueryDatasetPageSize)
)
```

#### Stack trace
```
Caused by: com.google.cloud.bigquery.BigQueryException: Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:299)
at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:778)
at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:775)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:774)
at .....
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://www.googleapis.com/bigquery/v2/projects/myproject/datasets/mydataset/tables/mytable?prettyPrint=false
{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"reason" : "rateLimitExceeded"
} ],
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"status" : "PERMISSION_DENIED"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:297)
... 44 common frames omitted
```

#### Any additional information below

I think the cause is the exception handling used in retry logic:

https://github.com/googleapis/java-bigquery/blob/f4e5fc59f4b9bc63c763ec1dc8b75f87defc9ced/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryBaseService.java#L29-L35

Afaiu that code makes abort retries in front of a `RuntimeException` and `BigQueryException` is so.

EDIT: Just i have observed the interceptor above could make BigqueryException retryable but 403 code is not in the list of retryable errors:

https://github.com/googleapis/java-bigquery/blob/f4e5fc59f4b9bc63c763ec1dc8b75f87defc9ced/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryException.java#L39-L41

I can see two alternatives if you think this should be fixed and the diagnostic is correct:
* open BigqueryOptions to allow users set a custom Exception handling
* Adds this case to the closed exception handler linked above

googleapis/java-bigquery#1498 asked to retry on a similiar error and was fixed but the return code was 400 in that case (instead 403)

Many thanks in advance

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.