DocumentInserter drops whole task batches when a payload key contains a dot
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 36
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
Hit this capturing real agent telemetry: a task whose payload embedded a CSV profile with a column literally named yr.doy. Mongo rejects . (and leading $) in field paths, and since mongodb_dao.insert_and_update_many_tasks writes tasks via $set upserts in a bulk_write, the whole batch raises BulkWriteError (code 16412) and every task in it is lost:
pymongo.errors.BulkWriteError: ... 'code': 16412,
'errmsg': "Invalid $set :: caused by :: FieldPath field names may not contain '.'.
Consider using $getField or $setField."
Could we sanitize/escape keys before building the update ops? Or is the expectation that the user does it? We currently work around it producer-side by escaping ./$ in payload keys before emission.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at mongodb_dao.insert_and_update_many_tasks, where payload keys become $set upserts in a bulk_write; reproduce the failure with a key such as yr.doy and inspect the resulting BulkWriteError. Establish the expected key encoding for MongoDB, then verify that a batch containing dotted or leading-$ keys completes without dropping tasks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100