googleapis / googleapis/google-cloud-rust
RetryPolicy decorator to treat `DeadlineExceeded` as retryable.
- Dominant language
- Rust
- Stars
- 955
- Forks
- 144
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 279
Description
For a gRPC delete, we got this error:
```
2025-08-17T22:48:34.900737Z ERROR ThreadId(77) storage_w1r3: message: delete error: Error { kind: Service(ServiceDetails { status_code: None, headers: Some({"grpc-server-stats-bin": "AADPw0I7AgAAAA", "endpoint-load-metrics-bin": "Me+AYTbGTFxASYtADbXjgdU/", "content-type": "application/grpc", "grpc-accept-encoding": "identity, deflate, gzip", "x-guploader-uploadid": "ABgVH89YKutJv9gugvxLaxAM0M4ERN0eNspL_uqVs6uzmMW53FhqoQnG0i-lhvZ9-olJNxbE", "content-length": "0", "date": "Sun, 17 Aug 2025 22:48:34 GMT"}), status: Status { code: DeadlineExceeded, message: "We encountered an internal error. Please try again.", details: [] } }), source: None }
2025-08-17T22:48:34.901265Z ERROR ThreadId(77) storage_w1r3: message: DELETE sample_builder = SampleBuilder { task: 426, relative: 5288.123763857s, iteration: 35599, start: Instant { tv_sec: 1304277, tv_nsec: 32046304 }, op: Delete, target_size: 20, object: "cUfCR8BJjmwre2HqbYYTjs37k6q48AT0" } error = Error { kind: Service(ServiceDetails { status_code: None, headers: Some({"grpc-server-stats-bin": "AADPw0I7AgAAAA", "endpoint-load-metrics-bin": "Me+AYTbGTFxASYtADbXjgdU/", "content-type": "application/grpc", "grpc-accept-encoding": "identity, deflate, gzip", "x-guploader-uploadid": "ABgVH89YKutJv9gugvxLaxAM0M4ERN0eNspL_uqVs6uzmMW53FhqoQnG0i-lhvZ9-olJNxbE", "content-length": "0", "date": "Sun, 17 Aug 2025 22:48:34 GMT"}), status: Status { code: DeadlineExceeded, message: "We encountered an internal error. Please try again.", details: [] } }), source: None }
```
The message suggests trying again, but we currently treat `DeadlineExceeeded` as non-retryable.
We should add decorators and a method to `RetryPolicyExt` to:
- [ ] Retry on `DeadlineExceeded`
- [x] Retry on client-side timeout errors (`gax::Error::is_timeout() == true`)
- [ ] Maybe a shortcut to retry on either
Contributor guide
Assessment
This issue has not been assessed yet.