datax在读取phoenix5时中文记录无法同步
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
以下是相关信息
JSON:
{
"job": {
"setting": {
"speed": {
"channel": 3,
"byte": 1048576
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "hbase20xsqlreader",
"parameter": {
"queryServerAddress": "http://192.168.1.68:8765",
"serialization": "PROTOBUF",
"schema": "XDZHU",
"table": "WNUSER",
"splitKey": "ID"
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"username": "xdzhut",
"password": "xdzhu123",
"column": [
"ID",
"ACCOUNT",
"PASSWD"
],
"connection": [
{
"table": [
"xxxx"
],
"jdbcUrl": "jdbc:mysql://192.168.2.162:3306/xdzhu"
}
]
}
}
}
],
"setting": {
"speed": {
"channel": "3"
}
}
}
}
phoenix数据:
0: jdbc:phoenix:> select * from xdzhu.wnuser;
+-----+----------+---------+
| ID | ACCOUNT | PASSWD |
+-----+----------+---------+
| 1 | 2 | 33 |
| 2 | 朱晓东 | 王振伟 |
+-----+----------+---------+
2 rows selected (0.014 seconds)
日志
INSERT INTO %s (ID,ACCOUNT,PASSWD) VALUES(?,?,?)
], which jdbcUrl like:[jdbc:mysql://192.168.2.162:3306/xdzhu?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2021-01-19 13:50:15.606 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2021-01-19 13:50:15.607 [job-0] INFO JobContainer - DataX Reader.Job [hbase20xsqlreader] do prepare work .
2021-01-19 13:50:15.607 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2021-01-19 13:50:15.607 [job-0] INFO JobContainer - jobContainer starts to do split ...
2021-01-19 13:50:15.607 [job-0] INFO JobContainer - Job set Channel-Number to 3 channels.
2021-01-19 13:50:15.608 [job-0] INFO HBase20SQLReaderHelper - Split accoring min and max value of splitColumn...
2021-01-19 13:50:15.624 [job-0] INFO HBase20SQLReaderHelper - Query SQL: select "ACCOUNT","ID","PASSWD" from "XDZHU"."WNUSER" where ('1' <= ID AND ID <= '2')
2021-01-19 13:50:15.624 [job-0] INFO JobContainer - DataX Reader.Job [hbase20xsqlreader] splits to [1] tasks.
2021-01-19 13:50:15.624 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2021-01-19 13:50:15.641 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2021-01-19 13:50:15.644 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2021-01-19 13:50:15.646 [job-0] INFO JobContainer - Running by standalone Mode.
2021-01-19 13:50:15.652 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2021-01-19 13:50:15.655 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2021-01-19 13:50:15.655 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2021-01-19 13:50:15.661 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2021-01-19 13:50:15.663 [0-0-0-reader] INFO HBase20xSQLReaderTask - Begin to read record by Sql: [select "ACCOUNT","ID","PASSWD" from "XDZHU"."WNUSER" where ('1' <= ID AND ID <= '2')
] {}.
2021-01-19 13:50:15.684 [0-0-0-reader] INFO HBase20xSQLReaderTask - Finished read record by Sql: [select "ACCOUNT","ID","PASSWD" from "XDZHU"."WNUSER" where ('1' <= ID AND ID <= '2')
] {}.
2021-01-19 13:50:16.261 [0-0-0-writer] WARN CommonRdbmsWriter$Task - 回滚此次写入, 采用每次写入一行方式提交. 因为:Incorrect integer value: '???' for column 'id' at row 2
2021-01-19 13:50:16.434 [0-0-0-writer] ERROR StdoutPluginCollector -
java.sql.SQLException: Incorrect integer value: '???' for column 'id' at row 1
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1199) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doOneInsert(CommonRdbmsWriter.java:382) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doBatchInsert(CommonRdbmsWriter.java:362) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWriteWithConnection(CommonRdbmsWriter.java:297) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWrite(CommonRdbmsWriter.java:319) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter$Task.startWrite(MysqlWriter.java:78) [mysqlwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) [datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_262]
2021-01-19 13:50:16.437 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据:
{"exception":"Incorrect integer value: '???' for column 'id' at row 1","record":[{"byteSize":3,"index":0,"rawData":"朱晓东","type":"STRING"},{"byteSize":1,"index":1,"rawData":"2","type":"STRING"},{"byteSize":3,"index":2,"rawData":"王振伟","type":"STRING"}],"type":"writer"}
2021-01-19 13:50:16.462 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[802]ms
2021-01-19 13:50:16.463 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2021-01-19 13:50:25.660 [job-0] INFO StandAloneJobContainerCommunicator - Total 2 records, 11 bytes | Speed 1B/s, 0 records/s | Error 1 records, 7 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2021-01-19 13:50:25.660 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2021-01-19 13:50:25.661 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do post work.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the provided hbase20xsqlreader-to-mysqlwriter job with the logged Chinese records. Trace the values from HBase20xSQLReaderTask into CommonRdbmsWriter$Task.doBatchInsert and doOneInsert, using the stack trace and dirty-record output as checkpoints. Done means Chinese values are written correctly to MySQL without the reported conversion error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100