GoogleCloudPlatform / GoogleCloudPlatform/DataflowTemplates
[Feature Request]: Expose read splitting options (bucketAuto / numSplits) in the MongoDB to BigQuery template
- Dominant language
- Java
- Stars
- 1.3k
- Forks
- 1.1k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 60
Description
### Related Template(s)
MongoDB to BigQuery
### What feature(s) are you requesting?
The MongoDB to BigQuery batch template currently reads the source collection as a single, unsplittable source. It calls `MongoDbIO.read()` with the defaults (`numSplits=0`, `bucketAuto=false`), and with those defaults Beam's `MongoDbIO.split()` [returns a single source](https://github.com/apache/beam/blob/v2.75.0/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java#L510), so the read phase never parallelizes no matter how many workers the job has. For large collections (hundreds of millions of documents and up) the read phase becomes the hard bottleneck of the whole job.
We hit this while backfilling a large MongoDB Atlas collection with this template. No matter how many workers we gave the job, the read stayed on a single reader: the Atlas cluster had plenty of headroom left, the Dataflow workers sat mostly idle, and end-to-end throughput was capped by that single read. Since the template exposes no parameter that affects read splitting, there was nothing to tune on the launch side.
I'd like the template to expose the splitting support Beam already ships as template parameters:
- `bucketAuto` (optional, default `false`): pass through to `MongoDbIO.read().withBucketAuto(...)`. `$bucketAuto` is a plain aggregation stage, so it also works on MongoDB Atlas where `splitVector` is not permitted. Keeping the default at `false` preserves the current behavior exactly.
- `numSplits` (optional, default `0`): pass through to `withNumSplits(...)` to control the number of read partitions (0 keeps the MongoDbIO default of 10 buckets when bucketAuto is enabled).
The change itself is limited to two `@TemplateParameter` definitions in `MongoDbToBigQueryOptions.MongoDbOptions` plus wiring them into the `MongoDbIO.read()` call in `MongoDbToBigQuery.run()`. We run a fork of this template with exactly this change in production, where it makes the difference between a single-reader job and one that actually scales with the worker pool.
There is also precedent in this repo. The newer MongoDB-to-MongoDB template already treats read-phase scalability as a first-class concern, and #4079 introduced a dynamic, type-aware bounds generator for splitting there. This request brings the MongoDB to BigQuery template in line with that direction.
One related note on the Beam side. For collections whose `_id` values are not ObjectIds (e.g. application-defined string ids), Beam's split-filter generation used to fail when splitting was enabled (apache/beam#39900). That fix has already been merged (apache/beam#39901) and is slated for Beam 2.77.0, so ObjectId-keyed collections benefit from `bucketAuto` immediately with the Beam version this repo is on today, and non-ObjectId collections are covered once the regular Beam dependency bump reaches 2.77.0. Happy to note this in the parameter's helpText.
I'd be glad to send a PR for this if the feature is acceptable.
Contributor guide
Research direction
Start with MongoDbToBigQueryOptions.MongoDbOptions and the MongoDbToBigQuery.run() entry point described in the issue, then inspect the existing MongoDbIO.read() call and nearby template parameters. Done means bucketAuto and numSplits are exposed with the stated defaults and passed through to the read operation without changing current default behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100