MongoEngine / MongoEngine/mongoengine

Cannot save documents with dots in a field key.

Open
#2,461 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.3k
Forks
1.2k
Avg merge
4h 41m
Merged PRs (30d)
11

Description

Issue https://github.com/MongoEngine/mongoengine/issues/1594 (related: https://github.com/MongoEngine/mongoengine/pull/2193) suggests that it has resolved the issue with having dots in a key name. However, this does't seem to be the case. Tested with mongoengine==0.22.1 and pymongo==3.11.2.

Although I'm raising it here I think that this is most likely a limitation of pymongo itself as https://docs.mongodb.com/manual/core/document/#field-names suggests that dots in key names aren't supported by official MongoDB drivers. However, I was just hoping to get a sanity check that https://github.com/MongoEngine/mongoengine/issues/1594 / https://github.com/MongoEngine/mongoengine/pull/2193 don't actually resolve the problem. I feel like in fields#DictField.validate that key_has_dot_or_dollar should be added back in (or possibly just a modified key_has_dot helper function)

For confirmation, I've tested with a $ in the key name and this works - it's just . that doesn't work.

A simple test case

import mongoengine
from mongoengine import fields, Document

mongoengine.connect(db="my_test_db")

class DocumentWithDots(Document):
    data = fields.DictField()

if __name__ == '__main__':
    DocumentWithDots(data={"t.est": "data"}).save()

Expected

document to be saved without issue

Actual

Traceback (most recent call last):
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/mongodot.py", line 14, in <module>
    DocumentWithDots(data={"t.est": "data"}).save()
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/mongoengine/document.py", line 398, in save
    object_id = self._save_create(doc, force_insert, write_concern)
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/mongoengine/document.py", line 463, in _save_create
    object_id = wc_collection.insert_one(doc).inserted_id
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/pymongo/collection.py", line 701, in insert_one
    session=session),
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/pymongo/collection.py", line 615, in _insert
    bypass_doc_val, session)
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/pymongo/collection.py", line 603, in _insert_one
    acknowledged, _insert_command, session)
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1498, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1384, in _retry_with_session
    return self._retry_internal(retryable, func, session, bulk)
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1416, in _retry_internal
    return func(session, sock_info, retryable)
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/pymongo/collection.py", line 598, in _insert_command
    retryable_write=retryable_write)
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/pymongo/pool.py", line 699, in command
    self._raise_connection_failure(error)
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/pymongo/pool.py", line 694, in command
    exhaust_allowed=exhaust_allowed)
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/pymongo/network.py", line 122, in command
    codec_options, ctx=compression_ctx)
  File "/home/ben/.virtualenvs/tmp-86f966e38f49f679/lib/python3.6/site-packages/pymongo/message.py", line 715, in _op_msg
    flags, command, identifier, docs, check_keys, opts)
bson.errors.InvalidDocument: key 't.est' must not contain '.'

Process finished with exit code 1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with fields#DictField.validate and the key_has_dot_or_dollar or key_has_dot helper mentioned in the report. Reproduce the failure using the provided DocumentWithDots example and check how MongoEngine and PyMongo handle the dotted key. Done means the supported behavior is confirmed and covered by an appropriate regression test or documented limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.