Add support for allowDiskUse (AggregationOptions) in MongoDbIO
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When a read is executed over a collection that exceed the memory limit of 104857600 an exception occurs. This is declared by mongodb and is possible to control the error passing a AggregationOptions allowDiskUse true so mongo can sort with disk usage.
This should be happen only when aggregations are added to read but now is happening even without aggregation at all.
Please let me know how can help with this improvement / bug.
!Screen Shot 2019-05-09 at 12.30.51.png!
```
PCollection> updateColls = p.apply("Reading Ops Collection: " + key, MongoDbIO
.read() .withUri(options.getMongoDBUri()) .withDatabase("local") .withCollection("oplog.rs") .withBucketAuto(true)
// .withQueryFn( // FindQuery.create().withFilters( // Filters.and( // Filters.gt("ts", ts.format(dtf)),
// Filters.eq("ns", options.getMongoDBDBName() + "" + key), // Filters.eq("op", "u") // ) // ) // //
AggregationQuery.create().withMongoDbPipeline(updatedDocsOplogAggregation) // ) )
```
Imported from Jira [BEAM-7256](https://issues.apache.org/jira/browse/BEAM-7256). Original Jira may contain additional context.
Reported by: jcornejo.
Contributor guide
Research direction
Locate MongoDbIO and trace the read path that creates or applies AggregationOptions; start by checking how reads behave with and without an aggregation. Done means allowDiskUse can be supported for aggregation reads while non-aggregation reads do not unexpectedly require it, with tests covering both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100