googleapis / googleapis/google-cloud-java
[java-bigquery] Bigquery: it should retry on BigQueryException: 403 Forbidden Exceeded rate limits
- Lenguaje dominante
- Java
- Estrellas
- 2.1k
- Forks
- 1.2k
- Merge medio
- 1 d 23 h
- PR fusionados (30 d)
- 154
Descripción
#### Environment details
1. OS type and version: linux (does not matter)
2. Java version: 1.8
3. version(s): 2.11.2 (But behaviour would be the same in newer versions afaiu, nos tested though)
#### Steps to reproduce
1. Make server return this response with status 403:
```json
{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"reason" : "rateLimitExceeded"
} ],
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"status" : "PERMISSION_DENIED"
}
```
2. Observe there are no retries
This have been tested mocking the server with wiremock and returning exactly that response, verifying the trace produced matched the production environment one
#### Code example
```java
// example
bigQueryService.listDatasets(
bigQueryConnection.getProjectId,
DatasetListOption.pageSize(Configuration.bigQueryDatasetPageSize)
)
```
#### Stack trace
```
Caused by: com.google.cloud.bigquery.BigQueryException: Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:299)
at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:778)
at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:775)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:774)
at .....
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://www.googleapis.com/bigquery/v2/projects/myproject/datasets/mydataset/tables/mytable?prettyPrint=false
{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"reason" : "rateLimitExceeded"
} ],
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"status" : "PERMISSION_DENIED"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:297)
... 44 common frames omitted
```
#### Any additional information below
I think the cause is the exception handling used in retry logic:
https://github.com/googleapis/java-bigquery/blob/f4e5fc59f4b9bc63c763ec1dc8b75f87defc9ced/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryBaseService.java#L29-L35
Afaiu that code makes abort retries in front of a `RuntimeException` and `BigQueryException` is so.
EDIT: Just i have observed the interceptor above could make BigqueryException retryable but 403 code is not in the list of retryable errors:
https://github.com/googleapis/java-bigquery/blob/f4e5fc59f4b9bc63c763ec1dc8b75f87defc9ced/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryException.java#L39-L41
I can see two alternatives if you think this should be fixed and the diagnostic is correct:
* open BigqueryOptions to allow users set a custom Exception handling
* Adds this case to the closed exception handler linked above
googleapis/java-bigquery#1498 asked to retry on a similiar error and was fixed but the return code was 400 in that case (instead 403)
Many thanks in advance
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.