MongoEngine / MongoEngine/mongoengine

QuerySetNoCache iterator protocol violation

Open
#1,316 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.3k
Forks
1.2k
Avg merge
4h 41m
Merged PRs (30d)
11

Description

I believe that the QuerySetNoCache object violates the iterator protocol because iter is not idempotent. I discovered this with code like the following:

from itertools import *
from mongoengine import *

class Foo(Document):
    pass

def chunks(iterator, n):
    it = iter(iterator)
    while True:
        r = tuple(islice(it, n))
        if not r:
            return
        yield r

connect('test')

Foo.objects.delete()
Foo().save()

qs = Foo.objects.no_cache()

for c in chunks(qs, 10):
    print c

This is an infinite loop because islice calls iter on its argument, but calling iter on the QuerySetNoCache rewinds the cursor. This can be repaired by changing the body of the __iter__ method to just return self, but that's not backwards compatible as it no longer clones the query set when iter is called on the original object multiple times. Perhaps the solution is something along the lines of

def __iter__(self):
    qs = self.clone()
    qs.rewind()
    while 1:
        yield qs.next()

This would fix the issue because the __iter__ method now never returns self, and the generator properly does. On the other hand, this looks odd to me, and I wouldn't submit it in a PR.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the QuerySetNoCache class and its iter implementation, then reproduce the issue with the chunks and islice example from the report. Done means repeated iteration remains compatible while nested iteration no longer rewinds the cursor or loops indefinitely; add or run focused iterator tests if the repository provides them.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.