MongoDBReader关于Job中split的切分问题
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
源为mongodb数据库时,使用多并发读取,Job在进行split切分算法时,并没有针对query条件进行数据过滤
而是读取全量数据通过 _id 进行切分,这样在collection中数据量较大时,切分会十分缓慢
真实场景:在源表数据为11亿条(存储空间约为4.5T)时,2channel切分耗时40min,3channel切分耗时90min
不针对query条件做过滤是有什么其它的考量吗
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the MongoDBReader split implementation and tracing how the query condition is handled during concurrent Job splitting. Compare the split behavior with and without the query filter, and determine whether scanning the full collection is intentional. Done means the issue has a documented decision or an implementation and validation path for avoiding unnecessary full-collection work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100