[Bug]: Mixed data types in _id field causing pipeline failures (SDK 2.53.0)
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
### Cause of the Problem
I'm experiencing a failure when reading a MongoDB collection on Atlas. The issue arises specifically when the pipeline attempts to process documents where the _id field contains mixed data types, such as strings and ObjectIds.
**Error Encountered:**
The pipeline fails during the phase where data is being read and chunked from MongoDB, with the following error message:
`TypeError: '>=' not supported between instances of 'str' and 'ObjectId'`
This error suggests that the pipeline is attempting to compare or range-partition _id fields of differing data types, leading to a type mismatch and ultimately causing the failure.
**Detailed Traceback:**
The failure seems to occur in the apache_beam/io/mongodbio.py file, specifically within the _get_auto_buckets and _range_is_not_splittable methods, indicating that the issue happens during the auto bucket generation for data chunking:
`File "/usr/local/lib/python3.8/site-packages/apache_beam/io/mongodbio.py", line 443, in _range_is_not_splittable
(isinstance(start_pos, str) and start_pos >= end_pos))`
**Suggested Solution:**
While the immediate workaround could be to ensure uniform data types for _id fields in MongoDB collections, this is not always feasible due to existing data and third-party data sources. Therefore, it would be beneficial if the MongoDB connector in Apache Beam could handle mixed data types in _id fields more gracefully, perhaps by treating all _id values as strings for the purpose of chunking or by providing a clear way to customize the chunking strategy.
**Steps to Reproduce:**
1. Set up a MongoDB collection with mixed data types in the _id field (e.g., some documents with string _id and others with ObjectId _id).
2. Create an Apache Beam pipeline that reads from this MongoDB collection.
3. Run the pipeline and observe the failure.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
Contributor guide
Assessment
This issue has not been assessed yet.