alibaba / alibaba/DataX

读取orc文件错误

Open
#1,995 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

1、环境:
HDFS 3.1.1.3.1
HIVE 3.0.0.3.1

hive表是ORC格式,

2、JOB配置:
{
"job": {
"setting": {
"speed": {
"channel": 3
}
},
"content": [
{
"reader": {
"name": "hdfsreader",
"parameter": {
"path": "/warehouse/tablespace/managed/hive/st.db/detail_article_book/date_no=2023/base_0000001/bucket_00000",
"defaultFS": "hdfs://10.80.1.100:8020/",
"column": ["*"],
"fileType": "orc",
"encoding": "UTF-8",
"fieldDelimiter": "\u0001",
"compress":"SNAPPY"
}

},
"writer": {
"name": "streamwriter",
"parameter": {
"print": true
}
}
}
]
}
}

3、问题:
直接从HDFS读数据后打印出来象下面这样,标红的那段不是表中的数据,不知从何而来,然后表中的数据被全部读到了一个字段中,就是标红后面的花括号括起来的那部分。
![image](https://github.com/alibaba/DataX/assets/13470396/081ec93c-7890-4d52-a3db-cfa9e24229a2)

各位有遇到过吗?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the hdfsreader entry point and reproduce the supplied job against the ORC path and configuration. Inspect how the reader interprets ORC records and maps columns, then verify the output contains only table values with each value in its own field.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.