mongodbreader切分channel的问题
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
[*com.luojilab.datax.plugin.reader.mongodbreader2.util.CollectionSplitUtil#doSplitCollection*](https://github.com/alibaba/DataX/blob/master/mongodbreader/src/main/java/com/alibaba/datax/plugin/reader/mongodbreader/util/CollectionSplitUtil.java#L141)
当没有splitVector授权时,会对全集数据做分页切分,无论是否有query参数,如下:
```java
int skipCount = chunkDocCount;
MongoCollection col = database.getCollection(collName);
for (int i = 0; i < splitPointCount; i++) {
Document doc = col.find().skip(skipCount).limit(chunkDocCount).first();
Object id = doc.get(KeyConstant.MONGO_PRIMARY_ID);
if (isObjectId) {
ObjectId oid = (ObjectId)id;
splitPoints.add(oid.toHexString());
} else {
splitPoints.add(id);
}
skipCount += chunkDocCount;
}
```
``` java
col.find(/*need query if configured*/).skip(skipCount).limit(chunkDocCount).first()
```
而这里的查询是不受speed参数控制的,很容易打满网卡
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in mongodbreader/src/main/java/com/alibaba/datax/plugin/reader/mongodbreader/util/CollectionSplitUtil.java at doSplitCollection, focusing on the no-splitVector path. Trace how the configured query and speed parameter are handled around the paginated find call. Done means splitting respects the query and does not issue unconstrained reads that bypass speed control.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100