alibaba / alibaba/DataX

mongodb增量用query reader数据时,当数据量很大时报错Sort exceeded memory limit

Open
#1,647 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

datax版本:
http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz

mongodb增量用query reader数据时,当数据量很大时报错报错详情:
com.mongodb.MongoQueryException: Query failed with error code 292 and error message 'PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.'

请问mongodbreader中有地方设置allowDiskUse:true吗?或者有其他方式解决分页排序问题不?
我在源码中未找到相关配置。

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting the MongoDBReader entry point and its query or pagination handling; the issue does not name specific source files or tests. Reproduce the large-result query and trace where sorting or aggregation is configured. Done means the reader can process the reported large dataset without the memory-limit failure, with a test or documented verification if the repository provides one.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.