MongoEngine / MongoEngine/mongoengine

Custom field that inherits from DictField results in strange behaviour when calling .from_json

Open
#2,692 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.3k
Forks
1.2k
Avg merge
4h 41m
Merged PRs (30d)
11

Description

I want to save pydantic models in my mongo database, so I made a custom field that inherits from DictField:

class BaseModelField(DictField):
    """
    A pydantic BaseModel field. Stores in the database using pydantic.BaseModel.dict().
    """

    def __init__(self, basemodel: Type[BaseModel], **kwargs):
        self._basemodel = basemodel
        super().__init__(**kwargs)

    def validate(self, value, *args, **kwargs):
        # Check correct type of basemodel
        if not isinstance(value, self._basemodel):
            self.error(f"Must be type {type(self._basemodel)}, not {type(value)}")
        # Further validation
        super().validate(value.dict())

    def to_mongo(self, value, *args, **kwargs):
        if isinstance(value, BaseModel):
            return super().to_mongo(value.dict(), *args, **kwargs)
        return super().to_mongo(value, *args, **kwargs)

    def to_python(self, value):
        if issubclass(type(value), self._basemodel):
            return value
        if isinstance(value, str):
            value = self._basemodel.parse_raw(value)
        if isinstance(value, dict):
            value = self._basemodel(**value)
        return value

When I make a document with this field, it can serialize to JSON fine, but when it de-serializes from json, it converts the basemodel field into a list of tuples.

from mongoengine.fields import DictField
from mongoengine.document import Document, EmbeddedDocument
from pydantic import BaseModel
from typing import Type

# Connect do db...


class BM(BaseModel):
    name: str


class Test(Document):
    data = BaseModelField(basemodel=BM)


t = Test(data=BM(name="Tom"))

The first time I call .to_json It works fine:

t.to_json()

Outputs:
'{"data": {"name": "Tom"}}'

However, If I call .from_json, It converts the dictionary to a list

rtn = Test.from_json(t.to_json())
print(rtn.data)

Outputs
[('name', 'Tom')]

And if I now get the .data attribute on the instance t, it has been modified:

t.data
[('name', 'Tom')]

I cannot work out what is happening. How can I serialize a pydantic model in this way into and from a mongo database?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the example using DictField, BaseModelField, Test.from_json, and the to_python path, then trace where the dictionary becomes a list of tuples and where the original instance is modified. Done means a JSON round trip preserves the pydantic model value instead of producing list tuples; add a regression test for that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.