django / django/new-features

Add Task.enqueue_on_commit() to Django's Tasks API

Open
#161 22 comments 13 reactions 0 assignees View on GitHub
Django Core Models/ORM Tasks
Dominant language
No language data
Stars
188
Forks
7
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow Django's Code of Conduct

### Feature Description

Add a convenience method to Django's built-in Tasks API for enqueueing a task only after the current database transaction commits.

Possible API:

```python
with transaction.atomic():
Thing.objects.create(num=1)
my_task.enqueue_on_commit(thing_num=1)
```

### Problem

Django's Tasks documentation already recommends wrapping task enqueueing in `transaction.on_commit()` when a task depends on database state created or updated in the current transaction:

```python
from functools import partial

from django.db import transaction

with transaction.atomic():
Thing.objects.create(num=1)
transaction.on_commit(partial(my_task.enqueue, thing_num=1))
```

This is correct, but it is verbose and easy to forget at write-to-queue boundaries. Enqueueing background work before commit is a common production race condition: the worker may start on another connection and fail to read the database row that the request just created.

In a real application using Django with Celery, every write-related enqueue boundary has to repeat this pattern:

```python
transaction.on_commit(
lambda: current_app.send_task(
TRANSCRIBE_VIDEO_TASK,
args=[video_id],
)
)
```

We use this pattern for transcription, indexing, transcript reindexing, account deletion, and evaluation tasks. Django's Tasks API could make the safe pattern explicit and discoverable.

### Request or proposal

proposal

### Additional Details

This would not change `Task.enqueue()` semantics. It would only provide a first-class API for the transaction-safe behavior Django already recommends.

The proposal is intentionally limited to the common case. `enqueue_on_commit()` would return `None`, since the real `TaskResult` is not available until the transaction commits.

This proposal also intentionally leaves out extra `transaction.on_commit()` options such as `using=` and `robust=`. Those names could conflict with normal task keyword arguments. Users who need those less common options can still call `transaction.on_commit()` directly.

An async counterpart such as `aenqueue_on_commit()` can be considered later, but the smallest useful version is the synchronous convenience method.

This is not about a specific task backend. The race exists for any backend that can run work outside the current database transaction.

### Implementation Suggestions

A minimal implementation could be:

```python
from functools import partial

from django.db import transaction

def enqueue_on_commit(self, *args, **kwargs):
transaction.on_commit(partial(self.enqueue, *args, **kwargs))
```

I'm happy to work on an implementation if the API is accepted.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.