MongoEngine / MongoEngine/mongoengine

MongoDB documents with non-ASCII keys break MongoEngine document instantiation

Open
#1,041 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Python 3.x
Dominant language
Python
Stars
4.3k
Forks
1.2k
Avg merge
4h 41m
Merged PRs (30d)
11

Description

We came across a DynamicEmbeddedDocument in our mongo instance that contained some accentuated keys, and ended up breaking the whole queryset retrieval through mongoengine.

As far as MongoDB's doc goes, it doesn't seem like there is any restriction on field names beyond the . and $ characters, and mongo(-shell) has no issue dealing with such (non-ASCII of UTF) fields.
http://docs.mongodb.org/manual/reference/limits/#Restrictions-on-Field-Names

Here is a minimalist example reproducing the issue (mongoengine 0.10.0), assuming a local DB is running:

$ virtualenv venv && venv/bin/pip install mongoengine ipython
$ venv/bin/ipython

# Document ready-to-paste in iPython
from mongoengine import connect, DynamicDocument

class MongoTest(DynamicDocument):
    '''Test main document'''
    meta = {
        'collection': 'mongo_test',
    }
    connect("test")

# '>' for commands issued in mongo-shell
# ':' for commands issued in iPython

> db.mongo_test.insert({"iamakey_/}+*#!": "value"})
WriteResult({ "nInserted" : 1 })

: MongoTest.objects().all()                                                                                      │
[<MongoTest: MongoTest object>]

> db.mongo_test.insert({"iamakey_/}+*#!helloUTFé": "value"})
WriteResult({ "nInserted" : 1 })

: all = MongoTest.objects().all()
: all
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)

# --truncated stack: iPython internals--

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/queryset/queryset.pyc in __repr__(self)
     56             return '.. queryset mid-iteration ..'
     57 
---> 58         self._populate_cache()
     59         data = self._result_cache[:REPR_OUTPUT_SIZE + 1]
     60         if len(data) > REPR_OUTPUT_SIZE:

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/queryset/queryset.pyc in _populate_cache(self)
     90             try:
     91                 for i in xrange(ITER_CHUNK_SIZE):
---> 92                     self._result_cache.append(self.next())
     93             except StopIteration:
     94                 self._has_more = False

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/queryset/base.pyc in next(self)
   1385             return self._get_as_pymongo(raw_doc)
   1386         doc = self._document._from_son(raw_doc,
-> 1387                                        _auto_dereference=self._auto_dereference, only_fields=self.only_fields)
   1388 
   1389         if self._scalar:

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/base/document.pyc in _from_son(cls, son, _auto_dereference, only_fields, created)
    729             data = dict((k, v)
    730                         for k, v in data.iteritems() if k in cls._fields)
--> 731         obj = cls(__auto_convert=False, _created=created, __only_fields=only_fields, **data)
    732         obj._changed_fields = changed_fields
    733         if not _auto_dereference:

...mongoengine_test/venv/local/lib/python2.7/site-packages/mongoengine/base/document.pyc in __init__(self, *args, **values)
    128             self._dynamic_lock = False
    129             for key, value in dynamic_data.iteritems():
--> 130                 setattr(self, key, value)
    131 
    132         # Flag initialised

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 22: ordinal not in range(128)

: all
[<MongoTest: MongoTest object>]

: len(all)
1

: all.count()
2

> db.mongo_test.find()
{ "_id" : ObjectId("558b51e14a55a1ecdbf608aa"), "iamakey_/}+*#!" : "value" }
{ "_id" : ObjectId("558b52154a55a1ecdbf608ab"), "iamakey_/}+*#!helloUTFé" : "value" }

The last instructions show that loading the queryset yields an encoding exception, but the cache is still populated, alas without the failing document.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the traceback entry points in mongoengine/base/document.py, especially _from_son and init, then follow how dynamic keys reach setattr. Reproduce the case with the two MongoTest documents and verify that queryset loading handles the non-ASCII key without raising UnicodeEncodeError or dropping the document; add regression coverage for that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.