apache / apache/beam

[Bug]: Mixed data types in _id field causing pipeline failures (SDK 2.53.0)

Open
#30,472 2 comments 0 reactions 0 assignees View on GitHub
bug P3 pinned python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

### Cause of the Problem

I'm experiencing a failure when reading a MongoDB collection on Atlas. The issue arises specifically when the pipeline attempts to process documents where the _id field contains mixed data types, such as strings and ObjectIds.

**Error Encountered:**

The pipeline fails during the phase where data is being read and chunked from MongoDB, with the following error message:

`TypeError: '>=' not supported between instances of 'str' and 'ObjectId'`

This error suggests that the pipeline is attempting to compare or range-partition _id fields of differing data types, leading to a type mismatch and ultimately causing the failure.

**Detailed Traceback:**

The failure seems to occur in the apache_beam/io/mongodbio.py file, specifically within the _get_auto_buckets and _range_is_not_splittable methods, indicating that the issue happens during the auto bucket generation for data chunking:

`File "/usr/local/lib/python3.8/site-packages/apache_beam/io/mongodbio.py", line 443, in _range_is_not_splittable
(isinstance(start_pos, str) and start_pos >= end_pos))`

**Suggested Solution:**

While the immediate workaround could be to ensure uniform data types for _id fields in MongoDB collections, this is not always feasible due to existing data and third-party data sources. Therefore, it would be beneficial if the MongoDB connector in Apache Beam could handle mixed data types in _id fields more gracefully, perhaps by treating all _id values as strings for the purpose of chunking or by providing a clear way to customize the chunking strategy.

**Steps to Reproduce:**

1. Set up a MongoDB collection with mixed data types in the _id field (e.g., some documents with string _id and others with ObjectId _id).
2. Create an Apache Beam pipeline that reads from this MongoDB collection.
3. Run the pipeline and observe the failure.

### Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

### Issue Components

- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.