MongoEngine / MongoEngine/mongoengine

mongoengine slow serialization of embedded documents with reference fields

Open
#1,367 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Performance
Dominant language
Python
Stars
4.3k
Forks
1.2k
Avg merge
4h 41m
Merged PRs (30d)
11

Description

I have a small DB with about 500 records. Using the following model:

class Structure(db.EmbeddedDocument):
    form = db.ReferenceField(Form, required = True)

    @property
    def short(self):    
        return {
             'form': self.form
        }

class Record(db.Document):
    form = db.ReferenceField(Form, required = True)
    structure = db.EmbeddedDocumentField(Structure)

    @property
    def short(self):    
        return {
            'form': self.form # Takes about 1s
            # 'structure': self.structure.short # Takes about 25s
        }

And performing:

start = time.clock()
records = Record.objects.select_related()
print ('Time: ', time.clock() - start)  
response = [i.short for i in records]
print ('Time: ', time.clock() - start)

I'm finding that the query and serialization time for 'form': self.form.short case takes 1.12s and 1.36s, respectively, however, for the 'structure': self.structure.short case it is 1.14s and 24.1s. The query times have not changed, however, the serialization time for an embedded document with a reference takes about 24 times longer...! Why such a big difference? I've also submitted this as a stackoverflow question in case someone wants some points. http://stackoverflow.com/questions/39088519/mongoengine-slow-serialization-of-embedded-documents-with-reference-fields

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Record.objects.select_related() query and the Structure.short and Record.short properties shown in the report. Reproduce both timing paths, compare direct and embedded reference serialization, and trace the source of the 24.1-second path. Done means the discrepancy is explained and the issue is addressed or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.