googleapis / googleapis/google-cloud-java

[java-bigquery] Bigquery: it should retry on BigQueryException: 403 Forbidden Exceeded rate limits

未关闭
#12,169 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
api: bigquery type: feature request
主要语言
Java
星标
2.1k
派生
1.2k
平均合并
1 天 23 小时
30 天内合并 PR
157

描述

#### Environment details

1. OS type and version: linux (does not matter)
2. Java version: 1.8
3. version(s): 2.11.2 (But behaviour would be the same in newer versions afaiu, nos tested though)

#### Steps to reproduce

1. Make server return this response with status 403:

```json
{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"reason" : "rateLimitExceeded"
} ],
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"status" : "PERMISSION_DENIED"
}
```

2. Observe there are no retries

This have been tested mocking the server with wiremock and returning exactly that response, verifying the trace produced matched the production environment one

#### Code example

```java
// example
bigQueryService.listDatasets(
bigQueryConnection.getProjectId,
DatasetListOption.pageSize(Configuration.bigQueryDatasetPageSize)
)
```

#### Stack trace
```
Caused by: com.google.cloud.bigquery.BigQueryException: Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:299)
at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:778)
at com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:775)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:774)
at .....
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET https://www.googleapis.com/bigquery/v2/projects/myproject/datasets/mydataset/tables/mytable?prettyPrint=false
{
"code" : 403,
"errors" : [ {
"domain" : "usageLimits",
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"reason" : "rateLimitExceeded"
} ],
"message" : "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas",
"status" : "PERMISSION_DENIED"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:297)
... 44 common frames omitted
```

#### Any additional information below

I think the cause is the exception handling used in retry logic:

https://github.com/googleapis/java-bigquery/blob/f4e5fc59f4b9bc63c763ec1dc8b75f87defc9ced/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryBaseService.java#L29-L35

Afaiu that code makes abort retries in front of a `RuntimeException` and `BigQueryException` is so.

EDIT: Just i have observed the interceptor above could make BigqueryException retryable but 403 code is not in the list of retryable errors:

https://github.com/googleapis/java-bigquery/blob/f4e5fc59f4b9bc63c763ec1dc8b75f87defc9ced/google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryException.java#L39-L41

I can see two alternatives if you think this should be fixed and the diagnostic is correct:
* open BigqueryOptions to allow users set a custom Exception handling
* Adds this case to the closed exception handler linked above

googleapis/java-bigquery#1498 asked to retry on a similiar error and was fixed but the return code was 400 in that case (instead 403)

Many thanks in advance

贡献指南

打开贡献指南

调研方向

从 google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryBaseService.java 和 BigQueryException.java 中链接的重试处理部分开始。复现 reason 为 rateLimitExceeded 的 WireMock 403 响应,然后跟踪它如何被重试逻辑分类。完成的标准是:该响应会被重试,同时不改变对不可重试 403 错误的处理方式。

由索引模型根据 Issue 内容生成。

评估

技术栈
google-cloud, java
领域
api, cloud
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。