High Memory Usage when Iterating over cursor with many document
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 97
- PR merge metrics
- No merged PRs in 30d
Description
# Bug
I have a use case where I need to query a collection that returns about ~1M object, which when my scripts is running I see memory usage upward of 1G. I thought this shouldn't happen as I am iterating over the cursor and not reading all object to memory at once.
### Current Behavior
```python
async for data in ENGINE.find(Data):
# do something with data
```
this will loads all object to memory because of `__aiter__` method in `AIOCursor` :
```python
async def __aiter__(self) -> AsyncGenerator[ModelType, None]:
if self._results is not None:
for res in self._results:
yield res
return
results = []
async for raw_doc in self._cursor:
instance = self._parse_document(raw_doc)
results.append(instance)
yield instance
self._results = results
```
### Expected behavior
shouldn't `__aiter__` yield each instance without caching it to memory since I'm _iterating_ over the cursor not reading all object to memory?
ex:
```python
async def __aiter__(self) -> AsyncGenerator[ModelType, None]:
async for raw_doc in self._cursor:
instance = self._parse_document(raw_doc)
yield instance
```
### Environment
- ODMantic version: 1.0.2
- MongoDB version: 7.0.9
- Pydantic infos (output of `python -c "import pydantic.utils; print(pydantic.utils.version_info())`):
```
pydantic version: 2.7.3
pydantic-core version: 2.18.4
pydantic-core build: profile=release pgo=true
install path: /home//.venv/lib/python3.12/site-packages/pydantic
python version: 3.12.3 (main, Apr 27 2024, 19:00:26) [GCC 9.4.0]
platform: Linux-5.4.0-182-generic-x86_64-with-glibc2.31
related packages: mypy-1.10.0 typing_extensions-4.12.1 pydantic-settings-2.3.1 fastapi-0.111.0
commit: unknown
```
**Additional context**
Curios why it needs to cache instance to a private variable when I'm iterating over the cursor.
Contributor guide
Research direction
Start at AIOCursor.__aiter__ and compare the cached _results path with iteration over _cursor. Reproduce the large async iteration described with ENGINE.find(Data), then verify that completed iteration does not retain every parsed instance while preserving any behavior that depends on cursor results being cached.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100