alibaba / alibaba/DataX

datax能支持两个带KB认证的hive吗

Open
#2,267 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

采集一个带KB认证的hive数据到另一个带KB认证的hive中。求大佬指导。
{
"job": {
"setting": {
"speed": {
"channel": 3
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "hdfsreader",
"parameter": {
"defaultFS": "hdfs://xxx.xxx.xx.4:8020",
"fileType": "orc",
"path": "/user/hive/warehouse/test.db/test67",
"fileName": "000000_0",
"fieldDelimiter": "\u0001",
"haveKerberos": true,
"kerberosKeytabFilePath": "/etc/keytab/hive.keytab",
"kerberosPrincipal": "hive/control-plane.minikube.internal@TEST.COM",
"nullFormat": "\\N",
"skipHeader": false,
"column": [
{
"index": "0",
"type": "Long"
}
]
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://xxx.xxx.xxx.5:8020",
"fileType": "orc",
"path": "/user/hive/warehouse/test.db/book",
"fileName": "cw",
"writeMode": "append",
"haveKerberos": true,
"kerberosKeytabFilePath": "/etc/keytab/hive.keytab",
"kerberosPrincipal": "hive/control-plane.minikube.internal@TEST.COM",
"fieldDelimiter": "\u0001",
"nullFormat": "\\N",
"column": [{
"name": "id",
"type": "int"
}]
}
}
}
]
}
}

因为一台机器只能认证一个hive,无法同时连两个KB认证的hive

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the hdfsreader and hdfswriter entry points in the supplied job configuration, focusing on how each handles Kerberos authentication. Determine whether two independently authenticated Hive/HDFS endpoints can be used in one job, and define the supported configuration and behavior for a successful transfer.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.