同步mysql2hdfs报错不支持SupportHiveDataType.DECIMAL
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
### 背景
使用datax集成mysql表数据到hdfs时,执行datax任务时报错不支持decimal类型。
mysql版本:8.0.25-1
datax版本:datax_v202205
hdfs版本:3.2.4
### 复现步骤
MySQL表结构及数据
```sql
DROP TABLE IF EXISTS `test_decimal_mysql`;
CREATE TABLE `test_decimal_mysql` (
`id` bigint NOT NULL,
`test_decimal` decimal(10,4) DEFAULT NULL COMMENT '测试decimal',
PRIMARY KEY (`id`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3 COMMENT='测试decimal';
INSERT INTO `test_decimal_mysql` (`id`, `test_decimal`) VALUES (1, 10.1516);
INSERT INTO `test_decimal_mysql` (`id`, `test_decimal`) VALUES (2, 20.3689);
```
datax job
```java
{
"content":[
{
"reader":{
"parameter":{
"password":"*************",
"connection":[
{
"querySql":[
"SELECT `id`, `test_decimal` FROM test_decimal_mysql"
],
"jdbcUrl":[
"jdbc:mysql://xxx.xxx.xxx.xxx:3306/testdb?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai&connectTimeout=5000"
]
}
],
"where":"",
"username":"root"
},
"name":"mysqlreader"
},
"writer":{
"parameter":{
"path":"/user/hive/warehouse/test_q.db/ods_test_decimal_mysql",
"fileName":"ods_test_decimal_mysql",
"compress":"NONE",
"hadoopConfig":{
"dfs.client.use.datanode.hostname":false,
"dfs.client.failover.proxy.provider.hacluster":"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
"dfs.ha.namenodes.hacluster":"namenode1,namenode2",
"dfs.namenode.rpc-address.hacluster.namenode2":"hdfs-namenodes.default:8020",
"dfs.nameservices":"hacluster",
"dfs.namenode.rpc-address.hacluster.namenode1":"hdfs-namenodes.default:8020"
},
"column":[
{
"name":"id",
"type":"BIGINT"
},
{
"name":"test_decimal",
"type":"DECIMAL"
}
],
"defaultFS":"hdfs://hdfs-namenodes.default:8020",
"writeMode":"append",
"fieldDelimiter":",",
"encoding":"utf-8",
"fileType":"orc"
},
"name":"hdfswriter"
}
}
],
"setting":{
"speed":{
"channel":1
}
}
}
```
### 报错信息
```java
com.alibaba.datax.common.exception.DataXException: Code:[Framework-13], Description:[DataX插件运行时出错, 具体原因请参看DataX运行结束时的错误诊断信息 .].
- java.lang.IllegalArgumentException: No enum constant com.alibaba.datax.plugin.writer.hdfswriter.SupportHiveDataType.DECIMAL
at java.lang.Enum.valueOf(Enum.java:238)
at com.alibaba.datax.plugin.writer.hdfswriter.SupportHiveDataType.valueOf(SupportHiveDataType.java:3)
at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.getColumnTypeInspectors(HdfsHelper.java:415)
at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.orcFileStartWrite(HdfsHelper.java:367)
at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Task.startWrite(HdfsWriter.java:367)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)
at java.lang.Thread.run(Thread.java:748)
- java.lang.IllegalArgumentException: No enum constant com.alibaba.datax.plugin.writer.hdfswriter.SupportHiveDataType.DECIMAL
at java.lang.Enum.valueOf(Enum.java:238)
at com.alibaba.datax.plugin.writer.hdfswriter.SupportHiveDataType.valueOf(SupportHiveDataType.java:3)
at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.getColumnTypeInspectors(HdfsHelper.java:415)
at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.orcFileStartWrite(HdfsHelper.java:367)
at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Task.startWrite(HdfsWriter.java:367)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)
at java.lang.Thread.run(Thread.java:748)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with SupportHiveDataType.java and the getColumnTypeInspectors and orcFileStartWrite paths in HdfsHelper.java, using the reported stack trace as the entry point. Run the supplied MySQL-to-HDFS DataX job and verify that the DECIMAL column is accepted and transferred successfully to the ORC output without the enum error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100