There seems to be a bug in datax's extraction of mongodb data. You need to set the batchSize for each find(). Will cause memory overflow。
Open
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
In MongodbReader, data is obtained through dbCursor = col.find(filter).iterator();, but the amount of data obtained in each batch is not set, which will cause memory overflow.
**Need to be modified like this**:
dbCursor = col.find(filter).**_batchSize(1000)_**.iterator();
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the MongodbReader implementation and inspect where dbCursor is created with col.find(filter).iterator(). Update that query path so each find uses a batch size of 1000 before iteration, then verify the MongoDB extraction behavior and confirm the cursor no longer fetches an unbounded batch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100