mongodbreader插件优化
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
1.原始的切片算法 col.find().skip(skipCount).limit(chunkDocCount).first()没有加过滤条件,
在对大数据量的mongodb表进行切片时,耗时太长。对mongodb的切片算法加上过滤条件
2.重写mongodb的反序列化,原有的方式对于很多mongodb类型处理的不好。
3.增加mongo的登录验证方式
4.新增了batchsize参数,批量读取mongodb表数据
5.设置read preference,优先读取副本
6.增加jsonType参数,可以将mongodb所有数据扫描json,而不是扫描固定的column字段(需求变化频繁的表)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the MongoDB reader plugin and reviewing its current slicing, deserialization, authentication, batch-size, read-preference, and JSON-reading behavior. Compare each of the six requested changes with the existing implementation and identify the relevant tests or fixtures. Done means the requested MongoDB options and data-handling behaviors work together and are covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100