apache / apache/opendal

new feature: introduce retry budget to prevent retry storm

Open
#7,376 0 comments 1 reaction 0 assignees View on GitHub
enhancement releases-note/feat
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Hi team, in terms for retry policy implemented in `RetryLayer`, currently we
- rely on `backon` for exponential backoff + jitter, with retry interval and attempt support
- if integrated with timeout layer, we could control the overall retry attempt upper bound, however many attempts we've tried internally

But we don't have any mechanism to prevent retry storm, basically when a storage backend is already overloaded we shouldn't attempt more retries to make service worse.

One way to achieve that is to introduce retry bucket
- when a request succeeds, we deposit the budget
- instead if one fails, we withdraw the budget
- retry can only happen when there's sufficient budget

As a reference, AWS S3 transfer manager have the implementation in their retry policy
- https://github.com/awslabs/aws-s3-transfer-manager-rs/blob/0226736b8ed3ffea5ad56ddfdc0ae08915ba3d6c/aws-sdk-s3-transfer-manager/src/operation/download/retry.rs#L19
- which internally uses `tower`'s [TpsBucket](https://github.com/tower-rs/tower/blob/master/tower/src/retry/budget/tps_budget.rs)

A somewhat relevant reference from google SRE book:

> Consider having a server-wide retry budget. For example, only allow 60 retries per minute in a process, and if the retry budget is exceeded, don’t retry; just fail the request.

Reference: https://sre.google/sre-book/addressing-cascading-failures/

### Problem and Solution

I want to avoid retry storm in production, introduce retry budget should solve or alleviate the problem.

### Additional Context

_No response_

### Are you willing to contribute to the development of this feature?

- [ ] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Start at the RetryLayer implementation and compare its retry-policy behavior with the referenced AWS S3 transfer-manager and tower TpsBucket implementations. Clarify the budget's scope, deposit and withdrawal rules, and interaction with backoff and timeout before implementing; done means retries stop when the budget is insufficient and the behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.