alibaba / alibaba/DataX

hdfs导出orc格式的hive表数据到mysql报错

Open
#717 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

hive版本是3.1,建了张测试表是orc格式的,尝试从hdfs导出数据到mysql,报错如下:com.alibaba.datax.common.exception.DataXException: Code:[HdfsReader-10], Description:[读取文件出错]. - 检查文件[hdfs://xxx/test_imsi_orc/delta_0000001_0000001_0000/_orc_acid_version]类型失败,目前支持ORC,SEQUENCE,RCFile,TEXT,CSV五种格式的文件,请检查您文件类型和文件是否正确。 - java.lang.IndexOutOfBoundsException
不知道是不是DataX的bug,
我的建表语句:
CREATE TABLE xxx.test_imsi_orc(
imsi string,
msisdn string
)row format delimited fields terminated by '|' STORED AS ORC;
job配置文件是这样的:
{
"job": {
"content": [
{
"reader": {
"name": "hdfsreader",
"parameter": {
"path": "hdfs://xx/test_imsi_orc",
"defaultFS": "hdfs://xx:8020",
"column": [
{
"index": 0,
"type": "string"
},
{
"index": 1,
"type": "string"
}
],
"fileType": "orc",
"encoding": "UTF-8",
"haveKerberos": true,
"kerberosKeytabFilePath": "xxxx",
"kerberosPrincipal": "xxxx"
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"column": ["imsi","msisdn"],
"connection": [
{
"jdbcUrl": "jdbc:mysql://xxxx/xxxx",
"table": ["test_imsi_orc"]
}
],
"password": "xxxx",
"preSql": [],
"session": [],
"username": "xxxx",
"writeMode": "insert"
}
}
}
],
"setting": {
"speed": {
"channel": "1"
}
}
}
}

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the HDFSReader export with the Hive 3.1 ORC table and the supplied job configuration. Inspect the HDFSReader handling of the test_imsi_orc path and its _orc_acid_version entry, then verify the MySQL writer path. Done means the ORC table exports successfully without the file-type or IndexOutOfBoundsException error.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.