MongoEngine / MongoEngine/mongoengine

upsert_one returns None even with primary read preference

Open
#1,964 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.3k
Forks
1.2k
Avg merge
4h 41m
Merged PRs (30d)
11

Description

It seems from the documentation that upsert_one should always return an object, but I keep getting back None on freshly created objects. E.g in pseudocode this always works in development, but fails in production:

new_object = ModelClass.objects(application=application_id).upsert_one(set_on_insert__date_created=now)
self.assertIsNotNone(new_object)

I assumed it might be related to the read preference in production (it is secondary), and tried the following:

new_object = ModelClass.                                    \
                objects(application=application_id).        \
                read_preference(Primary()).                 \
                upsert_one(set_on_insert__date_created=now)
self.assertIsNotNone(new_object)

Still doesn't work. The problem happens when line 564 gets triggered in queryset/base.py:

document = self._document.objects.with_id(atomic_update.upserted_id)

If - instead of self._document.objects.with_id - I re-do the query (and append the id), then it seems to take read preference into account, and finally works:

def _safe_upsert(app, **kwargs):
    def _query(**query_params):
        return ModelClass.                                     \
                    objects(**query_params).                   \
                    read_preference(read_pref.to_pymongo())

    query = _query(application=app.id)
    mlog.debug("Query class: %s", query.__class__)

    atomic_update = query.update(
                            multi=False,
                            upsert=True,
                            full_result=True,
                            **kwargs)
    if atomic_update.raw_result['updatedExisting']:
        return query.get()
    else:
        return _query(id=atomic_update.upserted_id)

Issue occurs with .no_cache() and using the default mongoengine QuerySet classes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in queryset/base.py at line 564, where upsert_one retrieves the upserted document through self._document.objects.with_id, and compare that path with the re-created query shown in the report. Reproduce the None result with no_cache(), the default QuerySet classes, and primary read preference; done means freshly upserted objects are returned consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.