alibaba / alibaba/DataX

There seems to be a bug in datax's extraction of mongodb data. You need to set the batchSize for each find(). Will cause memory overflow。

Open
#2,108 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

In MongodbReader, data is obtained through dbCursor = col.find(filter).iterator();, but the amount of data obtained in each batch is not set, which will cause memory overflow.
**Need to be modified like this**:
dbCursor = col.find(filter).**_batchSize(1000)_**.iterator();

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the MongodbReader implementation and inspect where dbCursor is created with col.find(filter).iterator(). Update that query path so each find uses a batch size of 1000 before iteration, then verify the MongoDB extraction behavior and confirm the cursor no longer fetches an unbounded batch.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.