googleapis / googleapis/google-cloud-go

bigquery: retry on error code:500

Open
#5,248 13 comments 5 reactions 1 assignee Claimed by @shollyman View on GitHub
api: bigquery type: question
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

**Client**

BigQuery

**Environment**

MacOS

**Go Environment**

go version go1.17.2 darwin/amd64

$ go env

**Code**

e.g.
```go
func retryableError(err error) bool {
if err == nil {
return false
}
if err == io.ErrUnexpectedEOF {
return true
}
// Special case due to http2: https://github.com/googleapis/google-cloud-go/issues/1793
// Due to Go's default being higher for streams-per-connection than is accepted by the
// BQ backend, it's possible to get streams refused immediately after a connection is
// started but before we receive SETTINGS frame from the backend. This generally only
// happens when we try to enqueue > 100 requests onto a newly initiated connection.
if err.Error() == "http2: stream closed" {
return true
}

switch e := err.(type) {
case *googleapi.Error:
// We received a structured error from backend.
var reason string
if len(e.Errors) > 0 {
reason = e.Errors[0].Reason
}
if e.Code == http.StatusServiceUnavailable || e.Code == http.StatusBadGateway || reason == "backendError" || reason == "rateLimitExceeded" {
return true
}
case *url.Error:
retryable := []string{"connection refused", "connection reset"}
for _, s := range retryable {
if strings.Contains(e.Error(), s) {
return true
}
}
case interface{ Temporary() bool }:
if e.Temporary() {
return true
}
}
// Unwrap is only supported in go1.13.x+
if e, ok := err.(interface{ Unwrap() error }); ok {
return retryableError(e.Unwrap())
}
return false
}

```

**Expected behavior**
When e.Code == http.StatusInternalServerError, according to the error message from BigQuery, it should retry.

**Actual behavior**
The error is not retried and return directly. Here's a sample message from our production env.
```
"error":"googleapi: Error 500: An internal error occurred and the request could not be completed. This is usually caused by a transient issue. Retrying the job with back-off as described in the BigQuery SLA should solve the problem: https://cloud.google.com/bigquery/sla. If the error continues to occur please contact support at https://cloud.google.com/support., internalError"
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.