alibaba / alibaba/DataX

csv抽取至hive,字段与数据对应关系不正确

Open
#340 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

csv数据:
![image](https://user-images.githubusercontent.com/39817883/58072388-207e2f80-7bd2-11e9-9944-f3ceeef77db9.png)

实际导入数据:
![image](https://user-images.githubusercontent.com/39817883/58072458-51f6fb00-7bd2-11e9-93bc-4e738f3e1ac7.png)
datax的json配置文件:
{
"job": {
"setting": {
"speed": {
"channel": 32
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [{
"reader": {
"name": "txtfilereader",
"parameter": {
"path": ["/home/haotian//measurements.csv"],
"encoding": "UTF-8",
"column": [{"index":0,"type":"string"},{"index":6,"type":"string"},{"index":8,"type":"string"},{"index":1,"type":"string"},{"index":12,"type":"string"},{"index":5,"type":"string"},{"index":2,"type":"string"},{"index":7,"type":"string"},{"index":10,"type":"string"},{"index":9,"type":"string"},{"index":4,"type":"string"},{"index":11,"type":"string"},{"index":3,"type":"string"}],
"fieldDelimiter":","
}
},
"writer":
{
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://192.168.2.112:8020",
"fileType": "text",
"path": "/apps/hive/warehouse/odsdb.db",
"fileName": "xwj3",
"column":[{"name":"CapturedTime","type":"string"},{"name":"DeviceID","type":"string"},{"name":"Height","type":"string"},{"name":"Latitude","type":"string"},{"name":"LoaderID","type":"string"},{"name":"LocationName","type":"string"},{"name":"Longitude","type":"string"},{"name":"MD5Sum","type":"string"},{"name":"Radiation","type":"string"},{"name":"Surface","type":"string"},{"name":"Unit","type":"string"},{"name":"UploadedTime","type":"string"},{"name":"Value","type":"string"}],
"writeMode": "append",
"fieldDelimiter": "\t"
}
}

}]
}
}

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the supplied CSV-to-Hive job with the shown txtfilereader and hdfswriter configuration, then inspect how the reader column indexes are paired with writer columns. Done means the imported fields match the intended CSV values and the mapping is covered by a reproducible verification.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.