Add the support of Lateral joins to django ORM
- Dominant language
- No language data
- Stars
- 188
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow Django's Code of Conduct
### Feature Description
When using `annotate()` with `Subquery()` expressions, Django inlines the full subquery SQL each time the annotation is referenced (e.g., in other annotations or in filters).
If a subquery annotation is reused multiple times, the database evaluates the identical subquery repeatedly instead of computing it once and reusing the result.
This leads to unnecessary work at the database level and can cause significant performance degradation for complex queries.
### Problem
```py
Product.objects.annotate(
on_stock=Subquery(
Stock.objects.filter(
product_id=OuterRef('id'),
).values('product_id').annotate(
on_stock=Sum('quantity'),
).values('on_stock').order_by(),
output_field=DecimalField(),
),
outgoing=Subquery(
OrderItem.objects.filter(
product_id=OuterRef('id'),
).values('product_id').annotate(
outgoing=Sum('quantity'),
).values('outgoing').order_by(),
output_field=DecimalField(),
),
available=F('on_stock') - F('outgoing'),
).filter(
available__gt=0,
).values('id', 'on_stock', 'outgoing', 'available')
```
Simplified SQL
```sql
SELECT product.id,
(SELECT SUM(...) FROM stock WHERE ...) AS on_stock,
(SELECT SUM(...) FROM order_item WHERE ...) AS outgoing,
((SELECT SUM(...) FROM stock WHERE ...)
- (SELECT SUM(...) FROM order_item WHERE ...)) AS available
FROM product
WHERE ((SELECT SUM(...) FROM stock WHERE ...)
- (SELECT SUM(...) FROM order_item WHERE ...)) > 0
```
Each subquery is inlined three times:
- In `SELECT`
- In the available calculation
- In the `WHERE` clause
Resulting in:
- 3 evaluations of `on_stock`
- 3 evaluations of outgoing
- 6 subquery executions per row instead of 2
Identical subquery annotations should be computed once per row and reused wherever referenced.
On PostgreSQL, this can be achieved using `LATERAL` joins:
```sql
SELECT product.id,
on_stock,
outgoing,
on_stock - outgoing AS available
FROM product
LEFT JOIN LATERAL (
SELECT SUM(quantity) AS on_stock
FROM stock
WHERE stock.product_id = product.id
) AS on_stock ON TRUE
LEFT JOIN LATERAL (
SELECT SUM(quantity) AS outgoing
FROM order_item
WHERE order_item.product_id = product.id
) AS outgoing ON TRUE
WHERE (on_stock - outgoing) > 0
```
This approach evaluates each subquery only once per row and allows reuse in both `SELECT` and `WHERE`.
The ORM duplicates the subquery SQL every time the annotation is referenced, resulting in repeated evaluation by the database engine.
Performance impact becomes significant when:
- Subqueries are complex
- Multiple derived annotations reuse the same subqueries
- Datasets are large
- Queries involve multiple chained annotations
### Request or proposal
proposal
### Additional Details
_No response_
### Implementation Suggestions
An ORM-level optimization could:
- Detect reusable Subquery annotations
- Convert them into `LATERAL` joins (where supported)
- Reuse computed aliases in `SELECT` and `WHERE`
This could be implemented as:
- A backend-specific optimization (e.g., PostgreSQL)
- Or a generalized mechanism where supported
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.