alibaba / alibaba/DataX

hdfsreader无法根据column获取数据【columnIndexMax 小于0】

Open
#1,327 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

**目的:根据hive列名获取数据
问题:hive无法根据列获取数据
版本:目前master分支拉取最新版本代码,hive存储 user表,存在id,name,age三列
{"type": "string","value": "id"} 配置的value值是对应hive表的列名吗?
column:[“*”] 可以正常执行任务,但是和想要结果不同。
是我配置有问题还是其他原因,有人出现过这样问题吗?**

```
经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[HdfsReader-00], Description:[您配置的值不合法.]. - 请确认您所读取的列配置正确!columnIndexMax 小于0,column:[{"type":"string","value":"id"},{"type":"string","value":"name"},{"type":"string","value":"age"}]
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
at com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil.orcFileStartRead(DFSUtil.java:366)
at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Task.startRead(HdfsReader.java:269)
at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:57)
at java.lang.Thread.run(Thread.java:748)
```

```
{
"reader":
{
"name": "hdfsreader",
"parameter":
{
"path": "/user/hive/warehouse/xxx.db/user/*.orc",
"defaultFS": "hdfs://xxxx:xxx",
"column":
[
{
"type": "string",
"value": "id"
},
{
"type": "string",
"value": "name"
},
{
"type": "string",
"value": "age"
}
],
"fileType": "orc",
"encoding": "UTF-8",
"fieldDelimiter": ","
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the supplied HdfsReader stack trace and inspect plugin/reader/hdfsreader/DFSUtil.java, especially orcFileStartRead, then follow how HdfsReader.java passes the column configuration. Reproduce the job with the provided ORC schema and column names; done means the cause of columnIndexMax being negative and the supported column configuration are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.