alibaba / alibaba/DataX

datax从hdfs读取数据时,写入mysql报错 Column 'xxx' cannot be null

Open
#1,354 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

## mysql 建表语句:
```sql
CREATE TABLE `api_shop_report_full_1d` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT '主键id',
`shop_id` bigint(20) NOT NULL DEFAULT '0',
`shop_name` varchar(100) NOT NULL DEFAULT '',
`expire_time` varchar(20) NOT NULL DEFAULT '',
`spark_type` int(4) NOT NULL DEFAULT '0',
`bail` bigint(20) NOT NULL DEFAULT '0',
`uv` bigint(20) NOT NULL DEFAULT '0',
`gms` bigint(20) NOT NULL DEFAULT '0',
`dt` date NOT NULL DEFAULT '1970-01-01',
`main_category_id` bigint(20) NOT NULL DEFAULT '-1',
`main_sec_category_id` bigint(20) NOT NULL DEFAULT '-1',
PRIMARY KEY (`id`),
KEY `idx_dt_shop_id` (`dt`,`shop_id`) USING BTREE
) COMMENT='店铺报表'
```
## job.json
```json
{
"job" : {
"content" : [ {
"reader" : {
"name" : "hdfsreader",
"parameter" : {
"defaultFS" : "hdfs://xxxx",
"path" : "/usr/hive/warehouse/tmp.db/api_shop_report_full_1d_65abec896a4c49969bba39a6f0d29bba",
"column" : [ {
"index" : 0,
"type" : "Long"
}, {
"index" : 1,
"type" : "String"
}, {
"index" : 2,
"type" : "String"
}, {
"index" : 3,
"type" : "Long"
}, {
"index" : 4,
"type" : "Long"
}, {
"index" : 5,
"type" : "Long"
}, {
"index" : 6,
"type" : "Long"
}, {
"index" : 7,
"type" : "String"
}, {
"index" : 8,
"type" : "Long"
}, {
"index" : 9,
"type" : "Long"
}],
"fileType" : "orc"
}
},
"writer" : {
"name" : "mysqlwriter",
"parameter" : {
"writeMode" : "insert",
"username" : "root",
"password" : "123456",
"column" : [ "shop_id", "shop_name", "expire_time", "spark_type", "bail", "uv", "gms", "dt", "main_category_id", "main_sec_category_id" ],
"session" : [ "set session sql_mode='ANSI'" ],
"preSql" : [],
"connection" : [ {
"jdbcUrl" : "jdbc:mysql://127.0.0.1:3306/dp_coral_validate?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=UTF-8&useSSL=false",
"table" : [ "api_shop_report_full_1d" ]
} ]
}
}
} ],
"setting" : {
"speed" : {
"channel" : 1,
"byte" : 262144
}
}
}
}
```
## 导入数据
```sql
insert INTO api_shop_report_full_1d (shop_id, shop_name, expire_time, spark_type, bail, uv, gms, dt, main_category_id,
main_sec_category_id)
VALUES ('358865', 'test123', '2022-11-06 19:06:38', '1', '100000', '1', '0', '2022-04-04', null, null);
```
## 问题描述
main_category_id,main_sec_category_id字段都为null的情况下
当 DataX 导入如上一条数据的时候会报 ```Column 'main_category_id' cannot be null```,
但是当DataX批量导入多条如上数据时,能够成功插入,并不会报 Column 'main_category_id' cannot be null错误。
虽然数据导入成功了,但是main_category_id,main_sec_category_id 这两个字段数据是错误的,因为前面建表这两个字段是 not null default -1. 但是导入的数据中,这两个字段值皆为 0

## 错误信息
```log
2022-04-28 15:53:16.674 [0-0-0-writer] INFO DBUtil - execute sql:[set session sql_mode='ANSI']
2022-04-28 15:53:16.865 [0-0-0-writer] INFO CommonRdbmsWriter$Task - columnIndex=8, strValue=null, type = true
2022-04-28 15:53:16.865 [0-0-0-writer] INFO CommonRdbmsWriter$Task - columnIndex=9, strValue=null, type = true
2022-04-28 15:53:16.937 [0-0-0-writer] WARN CommonRdbmsWriter$Task - 回滚此次写入, 采用每次写入一行方式提交. 因为:Column 'main_category_id' cannot be null
java.sql.BatchUpdateException: Column 'main_category_id' cannot be null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_322]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_322]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_322]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_322]
at com.mysql.cj.util.Util.handleNewInstance(Util.java:192) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.util.Util.getInstance(Util.java:167) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.util.Util.getInstance(Util.java:174) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.jdbc.exceptions.SQLError.createBatchUpdateException(SQLError.java:224) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchedInserts(ClientPreparedStatement.java:755) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchInternal(ClientPreparedStatement.java:426) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.jdbc.StatementImpl.executeBatch(StatementImpl.java:796) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doBatchInsert(CommonRdbmsWriter.java:358) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWriteWithConnection(CommonRdbmsWriter.java:298) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWrite(CommonRdbmsWriter.java:320) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter$Task.startWrite(MysqlWriter.java:78) [mysqlwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) [classes/:na]
at java.lang.Thread.run(Thread.java:750) [na:1.8.0_322]
Caused by: java.sql.SQLIntegrityConstraintViolationException: Column 'main_category_id' cannot be null
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:117) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1092) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1040) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1347) ~[mysql-connector-java-8.0.22.jar:8.0.22]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchedInserts(ClientPreparedStatement.java:716) ~[mysql-connector-java-8.0.22.jar:8.0.22]
... 8 common frames omitted
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at CommonRdbmsWriter.java, especially Task.doBatchInsert, and trace the null handling used by the mysqlwriter shown in the stack trace. Reproduce the single-row and batch cases from job.json against the supplied MySQL table. Done means the behavior is consistent and nullable input does not incorrectly replace the schema default of -1.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.