ORNL / ORNL/flowcept

DocumentInserter drops whole task batches when a payload key contains a dot

Open
#352 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
36
Forks
19
PR merge metrics
No merged PRs in 30d

Description

Hit this capturing real agent telemetry: a task whose payload embedded a CSV profile with a column literally named yr.doy. Mongo rejects . (and leading $) in field paths, and since mongodb_dao.insert_and_update_many_tasks writes tasks via $set upserts in a bulk_write, the whole batch raises BulkWriteError (code 16412) and every task in it is lost:

pymongo.errors.BulkWriteError: ... 'code': 16412,
'errmsg': "Invalid $set :: caused by :: FieldPath field names may not contain '.'.
           Consider using $getField or $setField."

Could we sanitize/escape keys before building the update ops? Or is the expectation that the user does it? We currently work around it producer-side by escaping ./$ in payload keys before emission.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at mongodb_dao.insert_and_update_many_tasks, where payload keys become $set upserts in a bulk_write; reproduce the failure with a key such as yr.doy and inspect the resulting BulkWriteError. Establish the expected key encoding for MongoDB, then verify that a batch containing dotted or leading-$ keys completes without dropping tasks.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.