hdfs导出orc格式的hive表数据到mysql报错
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
hive版本是3.1,建了张测试表是orc格式的,尝试从hdfs导出数据到mysql,报错如下:com.alibaba.datax.common.exception.DataXException: Code:[HdfsReader-10], Description:[读取文件出错]. - 检查文件[hdfs://xxx/test_imsi_orc/delta_0000001_0000001_0000/_orc_acid_version]类型失败,目前支持ORC,SEQUENCE,RCFile,TEXT,CSV五种格式的文件,请检查您文件类型和文件是否正确。 - java.lang.IndexOutOfBoundsException
不知道是不是DataX的bug,
我的建表语句:
CREATE TABLE xxx.test_imsi_orc(
imsi string,
msisdn string
)row format delimited fields terminated by '|' STORED AS ORC;
job配置文件是这样的:
{
"job": {
"content": [
{
"reader": {
"name": "hdfsreader",
"parameter": {
"path": "hdfs://xx/test_imsi_orc",
"defaultFS": "hdfs://xx:8020",
"column": [
{
"index": 0,
"type": "string"
},
{
"index": 1,
"type": "string"
}
],
"fileType": "orc",
"encoding": "UTF-8",
"haveKerberos": true,
"kerberosKeytabFilePath": "xxxx",
"kerberosPrincipal": "xxxx"
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"column": ["imsi","msisdn"],
"connection": [
{
"jdbcUrl": "jdbc:mysql://xxxx/xxxx",
"table": ["test_imsi_orc"]
}
],
"password": "xxxx",
"preSql": [],
"session": [],
"username": "xxxx",
"writeMode": "insert"
}
}
}
],
"setting": {
"speed": {
"channel": "1"
}
}
}
}
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the HDFSReader export with the Hive 3.1 ORC table and the supplied job configuration. Inspect the HDFSReader handling of the test_imsi_orc path and its _orc_acid_version entry, then verify the MySQL writer path. Done means the ORC table exports successfully without the file-type or IndexOutOfBoundsException error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, mysql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100