GoogleCloudPlatform / GoogleCloudPlatform/DataflowTemplates

[Feature Request]: Expose read splitting options (bucketAuto / numSplits) in the MongoDB to BigQuery template

Open Beginner friendly
#4,193 0 comments 0 reactions 0 assignees View on GitHub
addition needs triage p2
Dominant language
Java
Stars
1.3k
Forks
1.1k
Avg merge
2d 20h
Merged PRs (30d)
60

Description

### Related Template(s)

MongoDB to BigQuery

### What feature(s) are you requesting?

The MongoDB to BigQuery batch template currently reads the source collection as a single, unsplittable source. It calls `MongoDbIO.read()` with the defaults (`numSplits=0`, `bucketAuto=false`), and with those defaults Beam's `MongoDbIO.split()` [returns a single source](https://github.com/apache/beam/blob/v2.75.0/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java#L510), so the read phase never parallelizes no matter how many workers the job has. For large collections (hundreds of millions of documents and up) the read phase becomes the hard bottleneck of the whole job.

We hit this while backfilling a large MongoDB Atlas collection with this template. No matter how many workers we gave the job, the read stayed on a single reader: the Atlas cluster had plenty of headroom left, the Dataflow workers sat mostly idle, and end-to-end throughput was capped by that single read. Since the template exposes no parameter that affects read splitting, there was nothing to tune on the launch side.

I'd like the template to expose the splitting support Beam already ships as template parameters:

- `bucketAuto` (optional, default `false`): pass through to `MongoDbIO.read().withBucketAuto(...)`. `$bucketAuto` is a plain aggregation stage, so it also works on MongoDB Atlas where `splitVector` is not permitted. Keeping the default at `false` preserves the current behavior exactly.
- `numSplits` (optional, default `0`): pass through to `withNumSplits(...)` to control the number of read partitions (0 keeps the MongoDbIO default of 10 buckets when bucketAuto is enabled).

The change itself is limited to two `@TemplateParameter` definitions in `MongoDbToBigQueryOptions.MongoDbOptions` plus wiring them into the `MongoDbIO.read()` call in `MongoDbToBigQuery.run()`. We run a fork of this template with exactly this change in production, where it makes the difference between a single-reader job and one that actually scales with the worker pool.

There is also precedent in this repo. The newer MongoDB-to-MongoDB template already treats read-phase scalability as a first-class concern, and #4079 introduced a dynamic, type-aware bounds generator for splitting there. This request brings the MongoDB to BigQuery template in line with that direction.

One related note on the Beam side. For collections whose `_id` values are not ObjectIds (e.g. application-defined string ids), Beam's split-filter generation used to fail when splitting was enabled (apache/beam#39900). That fix has already been merged (apache/beam#39901) and is slated for Beam 2.77.0, so ObjectId-keyed collections benefit from `bucketAuto` immediately with the Beam version this repo is on today, and non-ObjectId collections are covered once the regular Beam dependency bump reaches 2.77.0. Happy to note this in the parameter's helpText.

I'd be glad to send a PR for this if the feature is acceptable.

Contributor guide

Open the contributing guide

Research direction

Start with MongoDbToBigQueryOptions.MongoDbOptions and the MongoDbToBigQuery.run() entry point described in the issue, then inspect the existing MongoDbIO.read() call and nearby template parameters. Done means bucketAuto and numSplits are exposed with the stated defaults and passed through to the read operation without changing current default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
data-engineering, databases
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.