MongoEngine / MongoEngine/mongoengine
Mongoengine is very slow on large documents compared to native pymongo usage
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.3k
- Forks
- 1.2k
- Avg merge
- 4h 41m
- Merged PRs (30d)
- 11
Description
(See also this StackOverflow question)
I have the following mongoengine model:
class MyModel(Document):
date = DateTimeField(required = True)
data_dict_1 = DictField(required = False)
data_dict_2 = DictField(required = True)
In some cases the document in the DB can be very large (around 5-10MB), and the data_dict fields contain complex nested documents (dict of lists of dicts, etc...).
I have encountered two (possibly related) issues:
- When I run native
pymongo find_one()query, it returns within a second. When I runMyModel.objects.first()it takes 5-10 seconds. - When I query a single large document from the DB, and then access its field, it takes 10-20 seconds just to do the following:
m = MyModel.objects.first()
val = m.data_dict_1.get(some_key)
The data in the object does not contain any references to any other objects, so it is not an issue of objects dereferencing.
I suspect it is related to some inefficiency of the internal data representation of mongoengine, which affects the document object construction as well as fields access.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the comparison between native pymongo find_one() and MyModel.objects.first() using the large MyModel document described here. Measure document construction separately from access to data_dict_1, then compare the timings and identify the internal path responsible; done means the cause is isolated and a measured improvement or actionable limitation is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100