django / django/new-features

Add retry / backoff API to Tasks

Open
#142 6 comments 15 reactions 0 assignees View on GitHub
Tasks
Dominant language
No language data
Stars
188
Forks
7
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow Django's Code of Conduct

### Feature Description

Tasks should support being retried.

My suggestion is that configuration comes in 2 parts: A fixed number of retries, and a function which returns number of attempts to backoff time. If the backoff function returns `None`, the task isn't retried anymore. `.using` cannot be used to modify these attributes.

The `task_result` passed to `backoff` will be after modification based on the current run (eg updating `status`, adding error details to `.errors`, incrementing `attempts` etc. The raw exception will also be passed, to access attributes which wouldn't survive serialization. This can be used to change the backoff based on the exception raised (eg changing the backoff if the HTTP error is `429` vs `502`).

Example:

```python
def linear_backoff(exception: BaseException, task_result: TaskResult) -> int | None:
return 30

@task(max_attempts=10, backoff=linear_backoff)
def my_task():
pass

my_task.enqueue() # Run at most 10 times, with 30 seconds in between
```

#### Settings.py

```python
TASKS = {
"default": {
"BACKEND": "django.tasks.backends.immediate.ImmediateBackend",
"MAX_ATTEMPTS": 10,
"BACKOFF": lambda task_result: 30,
}
}
```

### Problem

Currently, Tasks don't support retries - if a task fails it will stay failed requiring manual intervention. For most scales, the ability to "retry" a task when it fails is useful.

### Request or proposal

proposal

### Additional Details

Open questions:

- Does the backoff behaviour need to be more complex? Is it necessary to know how long the task took last time or other attributes about the task?
- Should either `max_attempts` or `backoff` be modifiable at enqueue time?
- `max_attempts` will be persisted - changes to the code after tasks are enqueued are not persistent, but changes to `backoff` _will_ be affected. Is that problematic?
- Should Django include some backoff methods built in? eg a `linear_backoff`, `exponential_backoff` etc?

### Implementation Suggestions

Retry functionality will need to be opt-in - defaulting to only a single execution (which also makes this change additive and backwards compatible). As part of retries, there will need to be a way to configure backoff.

Tasks are retried only if they fail (raise an exception). Successful tasks cannot be manually retried (or re-enqueued) - instead separate tasks should be created.

Retries should not interfere with `run_after` - it should keep the original value. However, the implementation may be similar.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.