art049 / art049/odmantic

High Memory Usage when Iterating over cursor with many document

Open
#490 1 comment 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1.2k
Forks
97
PR merge metrics
No merged PRs in 30d

Description

# Bug

I have a use case where I need to query a collection that returns about ~1M object, which when my scripts is running I see memory usage upward of 1G. I thought this shouldn't happen as I am iterating over the cursor and not reading all object to memory at once.

### Current Behavior

```python
async for data in ENGINE.find(Data):
# do something with data
```

this will loads all object to memory because of `__aiter__` method in `AIOCursor` :

```python
async def __aiter__(self) -> AsyncGenerator[ModelType, None]:
if self._results is not None:
for res in self._results:
yield res
return
results = []
async for raw_doc in self._cursor:
instance = self._parse_document(raw_doc)
results.append(instance)
yield instance
self._results = results
```

### Expected behavior

shouldn't `__aiter__` yield each instance without caching it to memory since I'm _iterating_ over the cursor not reading all object to memory?

ex:

```python
async def __aiter__(self) -> AsyncGenerator[ModelType, None]:
async for raw_doc in self._cursor:
instance = self._parse_document(raw_doc)
yield instance
```

### Environment

- ODMantic version: 1.0.2
- MongoDB version: 7.0.9
- Pydantic infos (output of `python -c "import pydantic.utils; print(pydantic.utils.version_info())`):

```
pydantic version: 2.7.3
pydantic-core version: 2.18.4
pydantic-core build: profile=release pgo=true
install path: /home//.venv/lib/python3.12/site-packages/pydantic
python version: 3.12.3 (main, Apr 27 2024, 19:00:26) [GCC 9.4.0]
platform: Linux-5.4.0-182-generic-x86_64-with-glibc2.31
related packages: mypy-1.10.0 typing_extensions-4.12.1 pydantic-settings-2.3.1 fastapi-0.111.0
commit: unknown
```

**Additional context**

Curios why it needs to cache instance to a private variable when I'm iterating over the cursor.

Contributor guide

Open the contributing guide

Research direction

Start at AIOCursor.__aiter__ and compare the cached _results path with iteration over _cursor. Reproduce the large async iteration described with ENGINE.find(Data), then verify that completed iteration does not retain every parsed instance while preserving any behavior that depends on cursor results being cached.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.