alibaba / alibaba/DataX

hdfswriter 不支持HDFS的HA

Open
#105 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

"writer": {
"name": "hdfswriter",
"parameter": {
**"defaultFS": "hdfs://hacluster",**
"fileType": "text",
"path": "/tmp/datax",
"fileName": "sg_ssd_rtu_p",
"column": [
{
"name": "col1",
"type": "STRING"
},
{
"name": "col2",
"type": "STRING"
},
{
"name": "col3",
"type": "STRING"
}
],
"writeMode": "append",
"fieldDelimiter": ","
}
}

}
]
}

**"defaultFS": "hdfs://hacluster",**

这样写法的FS,DataX无法识别和使用。

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the hdfswriter configuration path and trace how defaultFS is handled when set to hdfs://hacluster. Reproduce the issue with the supplied HA-style configuration; done means the writer recognizes and uses that defaultFS successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.