alibaba / alibaba/DataX

mongodbreader切分channel的问题

Open
#279 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

[*com.luojilab.datax.plugin.reader.mongodbreader2.util.CollectionSplitUtil#doSplitCollection*](https://github.com/alibaba/DataX/blob/master/mongodbreader/src/main/java/com/alibaba/datax/plugin/reader/mongodbreader/util/CollectionSplitUtil.java#L141)

当没有splitVector授权时,会对全集数据做分页切分,无论是否有query参数,如下:
```java
int skipCount = chunkDocCount;
MongoCollection col = database.getCollection(collName);
for (int i = 0; i < splitPointCount; i++) {
Document doc = col.find().skip(skipCount).limit(chunkDocCount).first();
Object id = doc.get(KeyConstant.MONGO_PRIMARY_ID);
if (isObjectId) {
ObjectId oid = (ObjectId)id;
splitPoints.add(oid.toHexString());
} else {
splitPoints.add(id);
}
skipCount += chunkDocCount;
}
```
``` java
col.find(/*need query if configured*/).skip(skipCount).limit(chunkDocCount).first()
```
而这里的查询是不受speed参数控制的,很容易打满网卡

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in mongodbreader/src/main/java/com/alibaba/datax/plugin/reader/mongodbreader/util/CollectionSplitUtil.java at doSplitCollection, focusing on the no-splitVector path. Trace how the configured query and speed parameter are handled around the paginated find call. Done means splitting respects the query and does not issue unconstrained reads that bypass speed control.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.