alibaba / alibaba/DataX

DataX抽取数据,数据源MongoDB字段值为null抽取报错

Open
#2,143 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

【版本号】
DataX:1.5

【描述】
MongoDB作为数据源写入Oracle,当脚本读取MongoDB的字段值为null时,会报源端与目标端字段不匹配。

【报错信息】
```
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] Exception in thread "taskGroup-0" com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-00], Description:[您的配置错误.]. - 列配置信息有错误. 因为您配置的任务中,源头读取字段数:1 与 目的表要写入的字段数:2 不相等. 请检查您的配置并作出修改.
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWriteWithConnection(CommonRdbmsWriter.java:278)
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWrite(CommonRdbmsWriter.java:319)
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.writer.oraclewriter.OracleWriter$Task.startWrite(OracleWriter.java:88)
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)
2024-06-25 09:46:14 [AnalysisStatistics.analysisStatisticsLog-53] at java.lang.Thread.run(Thread.java:748)
```

【脚本如下】
```
{
"job": {
"setting": {
"speed": {
"channel": 3,
"byte": 1048576
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mongodbreader",
"parameter": {
"address": [
"xxx"
],
"userName": "xxx",
"userPassword": "xxx",
"dbName": "cc_logs",
"collectionName": "xxx",
"column": [
{
"name": "_id",
"type": "Object"
},
{
"name": "ACCESS_NUMBER_DESC",
"type": "String"
}
]
}
},
"writer": {
"name": "oraclewriter",
"parameter": {
"username": "xxx",
"password": "xxx",
"column": [
"ID",
"ACCESS_NUMBER_DESC"
],
"connection": [
{
"table": [
"xxx"
],
"jdbcUrl": "jdbc:oracle:thin:@//xxx:xxx/xxx "
}
]
}
}
}
]
}
}
```

【备注】
源端与目标端均可正常连接,且自测当MongoDB字段【ACCESS_NUMBER_DESC】不为NULL时,脚本可正常插入,一旦【ACCESS_NUMBER_DESC】中有值为NULL时就报错。
不知道和版本有没有关系?也尝试了用query的方法,但是好像没办法读取MongoDB的sql,不知道有没有大佬有遇到过类似的问题,是怎么解决的?还望不吝赐教,感谢!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure through the mongodbreader-to-oraclewriter configuration shown, using a document where ACCESS_NUMBER_DESC is null and comparing it with a non-null value. Trace the reported column-count error from the reader and writer components; done means null values preserve both configured columns and can be inserted into the target table.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.