hdfsreader无法根据column获取数据【columnIndexMax 小于0】
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
**目的:根据hive列名获取数据
问题:hive无法根据列获取数据
版本:目前master分支拉取最新版本代码,hive存储 user表,存在id,name,age三列
{"type": "string","value": "id"} 配置的value值是对应hive表的列名吗?
column:[“*”] 可以正常执行任务,但是和想要结果不同。
是我配置有问题还是其他原因,有人出现过这样问题吗?**
```
经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[HdfsReader-00], Description:[您配置的值不合法.]. - 请确认您所读取的列配置正确!columnIndexMax 小于0,column:[{"type":"string","value":"id"},{"type":"string","value":"name"},{"type":"string","value":"age"}]
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
at com.alibaba.datax.plugin.reader.hdfsreader.DFSUtil.orcFileStartRead(DFSUtil.java:366)
at com.alibaba.datax.plugin.reader.hdfsreader.HdfsReader$Task.startRead(HdfsReader.java:269)
at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:57)
at java.lang.Thread.run(Thread.java:748)
```
```
{
"reader":
{
"name": "hdfsreader",
"parameter":
{
"path": "/user/hive/warehouse/xxx.db/user/*.orc",
"defaultFS": "hdfs://xxxx:xxx",
"column":
[
{
"type": "string",
"value": "id"
},
{
"type": "string",
"value": "name"
},
{
"type": "string",
"value": "age"
}
],
"fileType": "orc",
"encoding": "UTF-8",
"fieldDelimiter": ","
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the supplied HdfsReader stack trace and inspect plugin/reader/hdfsreader/DFSUtil.java, especially orcFileStartRead, then follow how HdfsReader.java passes the column configuration. Reproduce the job with the provided ORC schema and column names; done means the cause of columnIndexMax being negative and the supported column configuration are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100