django / django/new-features

Add the support of Lateral joins to django ORM

Open
#122 4 comments 18 reactions 0 assignees View on GitHub
Django Core Models/ORM
Dominant language
No language data
Stars
188
Forks
7
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow Django's Code of Conduct

### Feature Description

When using `annotate()` with `Subquery()` expressions, Django inlines the full subquery SQL each time the annotation is referenced (e.g., in other annotations or in filters).

If a subquery annotation is reused multiple times, the database evaluates the identical subquery repeatedly instead of computing it once and reusing the result.

This leads to unnecessary work at the database level and can cause significant performance degradation for complex queries.

### Problem

```py
Product.objects.annotate(
on_stock=Subquery(
Stock.objects.filter(
product_id=OuterRef('id'),
).values('product_id').annotate(
on_stock=Sum('quantity'),
).values('on_stock').order_by(),
output_field=DecimalField(),
),
outgoing=Subquery(
OrderItem.objects.filter(
product_id=OuterRef('id'),
).values('product_id').annotate(
outgoing=Sum('quantity'),
).values('outgoing').order_by(),
output_field=DecimalField(),
),
available=F('on_stock') - F('outgoing'),
).filter(
available__gt=0,
).values('id', 'on_stock', 'outgoing', 'available')
```
Simplified SQL
```sql
SELECT product.id,
(SELECT SUM(...) FROM stock WHERE ...) AS on_stock,
(SELECT SUM(...) FROM order_item WHERE ...) AS outgoing,
((SELECT SUM(...) FROM stock WHERE ...)
- (SELECT SUM(...) FROM order_item WHERE ...)) AS available
FROM product
WHERE ((SELECT SUM(...) FROM stock WHERE ...)
- (SELECT SUM(...) FROM order_item WHERE ...)) > 0
```
Each subquery is inlined three times:
- In `SELECT`
- In the available calculation
- In the `WHERE` clause

Resulting in:
- 3 evaluations of `on_stock`
- 3 evaluations of outgoing
- 6 subquery executions per row instead of 2

Identical subquery annotations should be computed once per row and reused wherever referenced.

On PostgreSQL, this can be achieved using `LATERAL` joins:
```sql
SELECT product.id,
on_stock,
outgoing,
on_stock - outgoing AS available
FROM product
LEFT JOIN LATERAL (
SELECT SUM(quantity) AS on_stock
FROM stock
WHERE stock.product_id = product.id
) AS on_stock ON TRUE
LEFT JOIN LATERAL (
SELECT SUM(quantity) AS outgoing
FROM order_item
WHERE order_item.product_id = product.id
) AS outgoing ON TRUE
WHERE (on_stock - outgoing) > 0
```
This approach evaluates each subquery only once per row and allows reuse in both `SELECT` and `WHERE`.

The ORM duplicates the subquery SQL every time the annotation is referenced, resulting in repeated evaluation by the database engine.

Performance impact becomes significant when:
- Subqueries are complex
- Multiple derived annotations reuse the same subqueries
- Datasets are large
- Queries involve multiple chained annotations

### Request or proposal

proposal

### Additional Details

_No response_

### Implementation Suggestions

An ORM-level optimization could:
- Detect reusable Subquery annotations
- Convert them into `LATERAL` joins (where supported)
- Reuse computed aliases in `SELECT` and `WHERE`

This could be implemented as:
- A backend-specific optimization (e.g., PostgreSQL)
- Or a generalized mechanism where supported

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.