DataX抽取数据,数据源MongoDB字段值为null抽取报错
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
【版本号】
DataX:1.5
【描述】
MongoDB作为数据源写入Oracle,当脚本读取MongoDB的字段值为null时,会报源端与目标端字段不匹配。
【报错信息】
```
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] Exception in thread "taskGroup-0" com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-00], Description:[您的配置错误.]. - 列配置信息有错误. 因为您配置的任务中,源头读取字段数:1 与 目的表要写入的字段数:2 不相等. 请检查您的配置并作出修改.
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWriteWithConnection(CommonRdbmsWriter.java:278)
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWrite(CommonRdbmsWriter.java:319)
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.writer.oraclewriter.OracleWriter$Task.startWrite(OracleWriter.java:88)
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at java.lang.Thread.run(Thread.java:748)
```
【脚本如下】
```
{
"job": {
"setting": {
"speed": {
"channel": 3,
"byte": 1048576
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mongodbreader",
"parameter": {
"address": [
"xxx"
],
"userName": "xxx",
"userPassword": "xxx",
"dbName": "cc_logs",
"collectionName": "xxx",
"column": [
{
"name": "_id",
"type": "Object"
},
{
"name": "ACCESS_NUMBER_DESC",
"type": "String"
}
]
}
},
"writer": {
"name": "oraclewriter",
"parameter": {
"username": "xxx",
"password": "xxx",
"column": [
"ID",
"ACCESS_NUMBER_DESC"
],
"connection": [
{
"table": [
"xxx"
],
"jdbcUrl": "jdbc:oracle:thin:@//xxx:xxx/xxx "
}
]
}
}
}
]
}
}
```
【备注】
源端与目标端均可正常连接,且自测当MongoDB字段【ACCESS_NUMBER_DESC】不为NULL时,脚本可正常插入,一旦【ACCESS_NUMBER_DESC】中有值为NULL时就报错。
不知道和版本有没有关系?也尝试了用query的方法,但是好像没办法读取MongoDB的sql,不知道有没有大佬有遇到过类似的问题,是怎么解决的?还望不吝赐教,感谢!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the failure through the mongodbreader-to-oraclewriter configuration shown, using a document where ACCESS_NUMBER_DESC is null and comparing it with a non-null value. Trace the reported column-count error from the reader and writer components; done means null values preserve both configured columns and can be inserted into the target table.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100