apache / apache/beam

Add support for allowDiskUse (AggregationOptions) in MongoDbIO

Open
#19,503 1 comment 0 reactions 0 assignees View on GitHub
improvement io java mongodb P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When a read is executed over a collection that exceed the memory limit of 104857600 an exception occurs. This is declared by mongodb and is possible to control the error passing a AggregationOptions allowDiskUse true so mongo can sort with disk usage. 

This should be happen only when aggregations are added to read but now is happening even without aggregation at all. 

Please let me know how can help with this improvement /  bug.

 

!Screen Shot 2019-05-09 at 12.30.51.png!  
```

PCollection> updateColls = p.apply("Reading Ops Collection: " + key, MongoDbIO
.read() .withUri(options.getMongoDBUri()) .withDatabase("local") .withCollection("oplog.rs") .withBucketAuto(true)
// .withQueryFn( // FindQuery.create().withFilters( // Filters.and( // Filters.gt("ts", ts.format(dtf)),
// Filters.eq("ns", options.getMongoDBDBName() + "" + key), // Filters.eq("op", "u") // ) // ) // //
AggregationQuery.create().withMongoDbPipeline(updatedDocsOplogAggregation) // ) )

```

Imported from Jira [BEAM-7256](https://issues.apache.org/jira/browse/BEAM-7256). Original Jira may contain additional context.
Reported by: jcornejo.

Contributor guide

Open the contributing guide

Research direction

Locate MongoDbIO and trace the read path that creates or applies AggregationOptions; start by checking how reads behave with and without an aggregation. Done means allowDiskUse can be supported for aggregation reads while non-aggregation reads do not unexpectedly require it, with tests covering both cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.