alibaba / alibaba/DataX

mongodbreader插件优化

Open
#738 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

1.原始的切片算法 col.find().skip(skipCount).limit(chunkDocCount).first()没有加过滤条件,
在对大数据量的mongodb表进行切片时,耗时太长。对mongodb的切片算法加上过滤条件
2.重写mongodb的反序列化,原有的方式对于很多mongodb类型处理的不好。
3.增加mongo的登录验证方式
4.新增了batchsize参数,批量读取mongodb表数据
5.设置read preference,优先读取副本
6.增加jsonType参数,可以将mongodb所有数据扫描json,而不是扫描固定的column字段(需求变化频繁的表)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the MongoDB reader plugin and reviewing its current slicing, deserialization, authentication, batch-size, read-preference, and JSON-reading behavior. Compare each of the six requested changes with the existing implementation and identify the relevant tests or fixtures. Done means the requested MongoDB options and data-handling behaviors work together and are covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.