alibaba / alibaba/DataX

MongoDBReader关于Job中split的切分问题

Open
#2,188 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

源为mongodb数据库时,使用多并发读取,Job在进行split切分算法时,并没有针对query条件进行数据过滤
而是读取全量数据通过 _id 进行切分,这样在collection中数据量较大时,切分会十分缓慢
真实场景:在源表数据为11亿条(存储空间约为4.5T)时,2channel切分耗时40min,3channel切分耗时90min

不针对query条件做过滤是有什么其它的考量吗

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the MongoDBReader split implementation and tracing how the query condition is handled during concurrent Job splitting. Compare the split behavior with and without the query filter, and determine whether scanning the full collection is intentional. Done means the issue has a documented decision or an implementation and validation path for avoiding unnecessary full-collection work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.