googleapis / googleapis/google-cloud-rust

RetryPolicy decorator to treat `DeadlineExceeded` as retryable.

Open
#2,927 0 comments 0 reactions 0 assignees View on GitHub
type: feature request
Dominant language
Rust
Stars
955
Forks
144
Avg merge
1d 5h
Merged PRs (30d)
279

Description

For a gRPC delete, we got this error:

```
2025-08-17T22:48:34.900737Z ERROR ThreadId(77) storage_w1r3: message: delete error: Error { kind: Service(ServiceDetails { status_code: None, headers: Some({"grpc-server-stats-bin": "AADPw0I7AgAAAA", "endpoint-load-metrics-bin": "Me+AYTbGTFxASYtADbXjgdU/", "content-type": "application/grpc", "grpc-accept-encoding": "identity, deflate, gzip", "x-guploader-uploadid": "ABgVH89YKutJv9gugvxLaxAM0M4ERN0eNspL_uqVs6uzmMW53FhqoQnG0i-lhvZ9-olJNxbE", "content-length": "0", "date": "Sun, 17 Aug 2025 22:48:34 GMT"}), status: Status { code: DeadlineExceeded, message: "We encountered an internal error. Please try again.", details: [] } }), source: None }

2025-08-17T22:48:34.901265Z ERROR ThreadId(77) storage_w1r3: message: DELETE sample_builder = SampleBuilder { task: 426, relative: 5288.123763857s, iteration: 35599, start: Instant { tv_sec: 1304277, tv_nsec: 32046304 }, op: Delete, target_size: 20, object: "cUfCR8BJjmwre2HqbYYTjs37k6q48AT0" } error = Error { kind: Service(ServiceDetails { status_code: None, headers: Some({"grpc-server-stats-bin": "AADPw0I7AgAAAA", "endpoint-load-metrics-bin": "Me+AYTbGTFxASYtADbXjgdU/", "content-type": "application/grpc", "grpc-accept-encoding": "identity, deflate, gzip", "x-guploader-uploadid": "ABgVH89YKutJv9gugvxLaxAM0M4ERN0eNspL_uqVs6uzmMW53FhqoQnG0i-lhvZ9-olJNxbE", "content-length": "0", "date": "Sun, 17 Aug 2025 22:48:34 GMT"}), status: Status { code: DeadlineExceeded, message: "We encountered an internal error. Please try again.", details: [] } }), source: None }
```

The message suggests trying again, but we currently treat `DeadlineExceeeded` as non-retryable.

We should add decorators and a method to `RetryPolicyExt` to:

- [ ] Retry on `DeadlineExceeded`
- [x] Retry on client-side timeout errors (`gax::Error::is_timeout() == true`)
- [ ] Maybe a shortcut to retry on either

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.