alibaba / alibaba/DataX

datax在读取phoenix5时中文记录无法同步

Open
#945 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

以下是相关信息
JSON:
{
"job": {
"setting": {
"speed": {
"channel": 3,
"byte": 1048576
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "hbase20xsqlreader",
"parameter": {
"queryServerAddress": "http://192.168.1.68:8765",
"serialization": "PROTOBUF",
"schema": "XDZHU",
"table": "WNUSER",
"splitKey": "ID"

}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"username": "xdzhut",
"password": "xdzhu123",
"column": [
"ID",
"ACCOUNT",
"PASSWD"
],
"connection": [
{
"table": [
"xxxx"
],
"jdbcUrl": "jdbc:mysql://192.168.2.162:3306/xdzhu"
}
]
}
}
}
],
"setting": {
"speed": {
"channel": "3"
}
}
}
}

phoenix数据:
0: jdbc:phoenix:> select * from xdzhu.wnuser;
+-----+----------+---------+
| ID | ACCOUNT | PASSWD |
+-----+----------+---------+
| 1 | 2 | 33 |
| 2 | 朱晓东 | 王振伟 |
+-----+----------+---------+
2 rows selected (0.014 seconds)

日志
INSERT INTO %s (ID,ACCOUNT,PASSWD) VALUES(?,?,?)
], which jdbcUrl like:[jdbc:mysql://192.168.2.162:3306/xdzhu?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2021-01-19 13:50:15.606 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2021-01-19 13:50:15.607 [job-0] INFO JobContainer - DataX Reader.Job [hbase20xsqlreader] do prepare work .
2021-01-19 13:50:15.607 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2021-01-19 13:50:15.607 [job-0] INFO JobContainer - jobContainer starts to do split ...
2021-01-19 13:50:15.607 [job-0] INFO JobContainer - Job set Channel-Number to 3 channels.
2021-01-19 13:50:15.608 [job-0] INFO HBase20SQLReaderHelper - Split accoring min and max value of splitColumn...
2021-01-19 13:50:15.624 [job-0] INFO HBase20SQLReaderHelper - Query SQL: select "ACCOUNT","ID","PASSWD" from "XDZHU"."WNUSER" where ('1' <= ID AND ID <= '2')
2021-01-19 13:50:15.624 [job-0] INFO JobContainer - DataX Reader.Job [hbase20xsqlreader] splits to [1] tasks.
2021-01-19 13:50:15.624 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2021-01-19 13:50:15.641 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2021-01-19 13:50:15.644 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2021-01-19 13:50:15.646 [job-0] INFO JobContainer - Running by standalone Mode.
2021-01-19 13:50:15.652 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2021-01-19 13:50:15.655 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2021-01-19 13:50:15.655 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2021-01-19 13:50:15.661 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2021-01-19 13:50:15.663 [0-0-0-reader] INFO HBase20xSQLReaderTask - Begin to read record by Sql: [select "ACCOUNT","ID","PASSWD" from "XDZHU"."WNUSER" where ('1' <= ID AND ID <= '2')
] {}.
2021-01-19 13:50:15.684 [0-0-0-reader] INFO HBase20xSQLReaderTask - Finished read record by Sql: [select "ACCOUNT","ID","PASSWD" from "XDZHU"."WNUSER" where ('1' <= ID AND ID <= '2')
] {}.
2021-01-19 13:50:16.261 [0-0-0-writer] WARN CommonRdbmsWriter$Task - 回滚此次写入, 采用每次写入一行方式提交. 因为:Incorrect integer value: '???' for column 'id' at row 2
2021-01-19 13:50:16.434 [0-0-0-writer] ERROR StdoutPluginCollector -
java.sql.SQLException: Incorrect integer value: '???' for column 'id' at row 1
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1199) ~[mysql-connector-java-5.1.34.jar:5.1.34]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doOneInsert(CommonRdbmsWriter.java:382) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doBatchInsert(CommonRdbmsWriter.java:362) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWriteWithConnection(CommonRdbmsWriter.java:297) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWrite(CommonRdbmsWriter.java:319) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter$Task.startWrite(MysqlWriter.java:78) [mysqlwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) [datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_262]
2021-01-19 13:50:16.437 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据:
{"exception":"Incorrect integer value: '???' for column 'id' at row 1","record":[{"byteSize":3,"index":0,"rawData":"朱晓东","type":"STRING"},{"byteSize":1,"index":1,"rawData":"2","type":"STRING"},{"byteSize":3,"index":2,"rawData":"王振伟","type":"STRING"}],"type":"writer"}
2021-01-19 13:50:16.462 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[802]ms
2021-01-19 13:50:16.463 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2021-01-19 13:50:25.660 [job-0] INFO StandAloneJobContainerCommunicator - Total 2 records, 11 bytes | Speed 1B/s, 0 records/s | Error 1 records, 7 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2021-01-19 13:50:25.660 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2021-01-19 13:50:25.661 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do post work.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the provided hbase20xsqlreader-to-mysqlwriter job with the logged Chinese records. Trace the values from HBase20xSQLReaderTask into CommonRdbmsWriter$Task.doBatchInsert and doOneInsert, using the stack trace and dirty-record output as checkpoints. Done means Chinese values are written correctly to MySQL without the reported conversion error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.