Django's querysets slice in a strange way when the data is already cached
- Dominant language
- No language data
- Stars
- 188
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow Django's Code of Conduct
### Feature Description
Let evaluated `QuerySet`s when sliced return a new `QuerySet` with a cache that is sliced, not a list.
### Problem
Django's `QuerySet`s typically make a *new* queryset when applying operations on it, like slicing, filtering, annotating, etc.
But slicing seems to perform a bit odd when the *original* queryset is already loaded into memory.
Indeed, imagine the following `QuerySet`:
```python
from django.contrib.auth.models import User
original = User.objects.all()
sliced = original[:10] # type of sliced is QuerySet
```
but now imagine we somehow first consume the original:
```python
from django.contrib.auth.models import User
original = User.objects.all()
list(original) # makes a query and fetches the items in the cache
sliced = original[:10] # type of sliced is list
```
Now most of the time methods *receive* a `QuerySet` not even knowing if the `QuerySet` got evaluated in the first place or not, so it is difficult/impossible to tell if we will receive a `list` or `QuerySet`.
The fact that the `QuerySet` already was consumed can help: we can make a new `QuerySet` where we already populate the cache with the sliced cache of the `original`, so preventing an extra roundtrip to the database.
But by keeping it a queryset, a function that might not be aware of the fact that the item has been evaluated can still perform some options on it, like annotating, for example:
```python
from django.db.models import F
def slice_annotate(qs):
return qs[:10].annotate(pk2=F('pk'))
```
this will work if `qs` is not evaluated first, otherwise it will fail with an `AttributeError`, but at the time of writing the method, we do not know that.
### Request or proposal
proposal
### Additional Details
_No response_
### Implementation Suggestions
The current `QuerySet` class can probably be slightly rewritten, where we replace:
```python
def __getitem__(self, k):
# ...
if self._result_cache is not None:
return self._result_cache[k]
# ...
if isinstance(k, slice):
# ...
return list(qs)[:: k.step] if k.step else qs
# ...
```
by:
```python
def __getitem__(self, k):
# ...
if self._result_cache is not None:
if not isinstance(k, slice) or k.step:
# ... old behavior
return self._result_cache[k]
# ...
if isinstance(k, slice):
# ...
if k.step:
return list(qs)[:: k.step]
elif self._result_cache is not None:
qs._result_cache = self._result_cache[k]
return qs
```
that way we *have* a `QuerySet` that we can further inspect, change, etc. But we also have already cached the items in it.
It also simplifies the type reasoning for the programmer.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the current QuerySet class and its __getitem__ implementation, especially the _result_cache and slice handling described in the issue. Verify the behavior for evaluated and unevaluated querysets, then add coverage for sliced cached querysets and confirm the result remains a QuerySet with the sliced cache while stepped slices retain their existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- database
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100