Add retry / backoff API to Tasks
- Dominant language
- No language data
- Stars
- 188
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow Django's Code of Conduct
### Feature Description
Tasks should support being retried.
My suggestion is that configuration comes in 2 parts: A fixed number of retries, and a function which returns number of attempts to backoff time. If the backoff function returns `None`, the task isn't retried anymore. `.using` cannot be used to modify these attributes.
The `task_result` passed to `backoff` will be after modification based on the current run (eg updating `status`, adding error details to `.errors`, incrementing `attempts` etc. The raw exception will also be passed, to access attributes which wouldn't survive serialization. This can be used to change the backoff based on the exception raised (eg changing the backoff if the HTTP error is `429` vs `502`).
Example:
```python
def linear_backoff(exception: BaseException, task_result: TaskResult) -> int | None:
return 30
@task(max_attempts=10, backoff=linear_backoff)
def my_task():
pass
my_task.enqueue() # Run at most 10 times, with 30 seconds in between
```
#### Settings.py
```python
TASKS = {
"default": {
"BACKEND": "django.tasks.backends.immediate.ImmediateBackend",
"MAX_ATTEMPTS": 10,
"BACKOFF": lambda task_result: 30,
}
}
```
### Problem
Currently, Tasks don't support retries - if a task fails it will stay failed requiring manual intervention. For most scales, the ability to "retry" a task when it fails is useful.
### Request or proposal
proposal
### Additional Details
Open questions:
- Does the backoff behaviour need to be more complex? Is it necessary to know how long the task took last time or other attributes about the task?
- Should either `max_attempts` or `backoff` be modifiable at enqueue time?
- `max_attempts` will be persisted - changes to the code after tasks are enqueued are not persistent, but changes to `backoff` _will_ be affected. Is that problematic?
- Should Django include some backoff methods built in? eg a `linear_backoff`, `exponential_backoff` etc?
### Implementation Suggestions
Retry functionality will need to be opt-in - defaulting to only a single execution (which also makes this change additive and backwards compatible). As part of retries, there will need to be a way to configure backoff.
Tasks are retried only if they fail (raise an exception). Successful tasks cannot be manually retried (or re-enqueued) - instead separate tasks should be created.
Retries should not interfere with `run_after` - it should keep the original value. However, the implementation may be similar.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.