django / django/new-features

Django's querysets slice in a strange way when the data is already cached

Open
#184 7 comments 1 reaction 0 assignees View on GitHub
Django Core Models/ORM
Dominant language
No language data
Stars
188
Forks
7
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow Django's Code of Conduct

### Feature Description

Let evaluated `QuerySet`s when sliced return a new `QuerySet` with a cache that is sliced, not a list.

### Problem

Django's `QuerySet`s typically make a *new* queryset when applying operations on it, like slicing, filtering, annotating, etc.

But slicing seems to perform a bit odd when the *original* queryset is already loaded into memory.

Indeed, imagine the following `QuerySet`:

```python
from django.contrib.auth.models import User

original = User.objects.all()
sliced = original[:10] # type of sliced is QuerySet
```

but now imagine we somehow first consume the original:

```python
from django.contrib.auth.models import User

original = User.objects.all()
list(original) # makes a query and fetches the items in the cache

sliced = original[:10] # type of sliced is list
```

Now most of the time methods *receive* a `QuerySet` not even knowing if the `QuerySet` got evaluated in the first place or not, so it is difficult/impossible to tell if we will receive a `list` or `QuerySet`.

The fact that the `QuerySet` already was consumed can help: we can make a new `QuerySet` where we already populate the cache with the sliced cache of the `original`, so preventing an extra roundtrip to the database.

But by keeping it a queryset, a function that might not be aware of the fact that the item has been evaluated can still perform some options on it, like annotating, for example:

```python
from django.db.models import F

def slice_annotate(qs):
return qs[:10].annotate(pk2=F('pk'))
```

this will work if `qs` is not evaluated first, otherwise it will fail with an `AttributeError`, but at the time of writing the method, we do not know that.

### Request or proposal

proposal

### Additional Details

_No response_

### Implementation Suggestions

The current `QuerySet` class can probably be slightly rewritten, where we replace:

```python
def __getitem__(self, k):
# ...
if self._result_cache is not None:
return self._result_cache[k]
# ...
if isinstance(k, slice):
# ...
return list(qs)[:: k.step] if k.step else qs
# ...
```

by:

```python
def __getitem__(self, k):
# ...
if self._result_cache is not None:
if not isinstance(k, slice) or k.step:
# ... old behavior
return self._result_cache[k]
# ...
if isinstance(k, slice):
# ...
if k.step:
return list(qs)[:: k.step]
elif self._result_cache is not None:
qs._result_cache = self._result_cache[k]
return qs
```

that way we *have* a `QuerySet` that we can further inspect, change, etc. But we also have already cached the items in it.

It also simplifies the type reasoning for the programmer.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the current QuerySet class and its __getitem__ implementation, especially the _result_cache and slice handling described in the issue. Verify the behavior for evaluated and unevaluated querysets, then add coverage for sliced cached querysets and confirm the result remains a QuerySet with the sliced cache while stepped slices retain their existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
database
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.