new feature: introduce retry budget to prevent retry storm
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
Hi team, in terms for retry policy implemented in `RetryLayer`, currently we
- rely on `backon` for exponential backoff + jitter, with retry interval and attempt support
- if integrated with timeout layer, we could control the overall retry attempt upper bound, however many attempts we've tried internally
But we don't have any mechanism to prevent retry storm, basically when a storage backend is already overloaded we shouldn't attempt more retries to make service worse.
One way to achieve that is to introduce retry bucket
- when a request succeeds, we deposit the budget
- instead if one fails, we withdraw the budget
- retry can only happen when there's sufficient budget
As a reference, AWS S3 transfer manager have the implementation in their retry policy
- https://github.com/awslabs/aws-s3-transfer-manager-rs/blob/0226736b8ed3ffea5ad56ddfdc0ae08915ba3d6c/aws-sdk-s3-transfer-manager/src/operation/download/retry.rs#L19
- which internally uses `tower`'s [TpsBucket](https://github.com/tower-rs/tower/blob/master/tower/src/retry/budget/tps_budget.rs)
A somewhat relevant reference from google SRE book:
> Consider having a server-wide retry budget. For example, only allow 60 retries per minute in a process, and if the retry budget is exceeded, don’t retry; just fail the request.
Reference: https://sre.google/sre-book/addressing-cascading-failures/
### Problem and Solution
I want to avoid retry storm in production, introduce retry budget should solve or alleviate the problem.
### Additional Context
_No response_
### Are you willing to contribute to the development of this feature?
- [ ] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
Start at the RetryLayer implementation and compare its retry-policy behavior with the referenced AWS S3 transfer-manager and tower TpsBucket implementations. Clarify the budget's scope, deposit and withdrawal rules, and interaction with backoff and timeout before implementing; done means retries stop when the budget is insufficient and the behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100