DTStack / DTStack/chunjun

[Bug] [sync] 两个带kerberos认证的CDH-6.3.2集群之间同步数据报错

Open
#1,659 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.1k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar issues.

### What happened

![6bd49a4b28ec26a55291d214dd216c0](https://github.com/DTStack/chunjun/assets/29533681/ce497476-10fc-46a5-b7c5-81672a30043a)
以上是我的报错。

------------------------我是分割线---------------------
```json
{
"job": {
"content": [
{
"reader" : {
"parameter" : {
"path" : "hdfs://ns1/user/hive/warehouse/payment_i17_ods_phq.db/city_info/*",
"hadoopConfig" : {
"hadoop.user.name": "dataarchitecture_reader",
"dfs.ha.namenodes.ns1": "nn1,nn2",
"fs.defaultFS": "hdfs://ns1",
"dfs.namenode.rpc-address.ns1.nn2": "node104:8020",
"dfs.client.failover.proxy.provider.ns1": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
"dfs.namenode.rpc-address.ns1.nn1": "node105:8020",
"dfs.nameservices": "ns1",
"fs.hdfs.impl.disable.cache": "true",
"fs.hdfs.impl": "org.apache.hadoop.hdfs.DistributedFileSystem",
"hadoop.security.authentication": "Kerberos",
"dfs.namenode.kerberos.principal": "hdfs/_HOST@TEST.COM",
"dfs.datanode.kerberos.principal": "hdfs/_HOST@TEST.COM",
"yarn.resourcemanager.principal": "rm/_HOST@TEST.COM",
"dfs.namenode.kerberos.internal.spnego.principal": "HTTP/_HOST@TEST.COM",
"hadoop.security.authorization": "true",
"dfs.namenode.keytab.file": "/app/chunjun/kerberos/dataarchitecture_reader.keytab",
"java.security.krb5.conf": "/app/chunjun/krb5/krb5_cdh1.conf",
"useLocalFile": "true",
"principalFile": "/app/chunjun/kerberos/dataarchitecture_reader.keytab",
"principal": "dataarchitecture_reader/node105@TEST.COM",
"dfs.client.use.datanode.hostname":true,
"dfs.client.use.namenode.hostname":true,
"sun.security.krb5.disableReferrals":true
},
"column": [
{
"name":"city_id",
"type":"string"
},
{
"name":"city_name",
"type":"string"
}
],
"defaultFS" : "hdfs://ns1",
"fieldDelimiter" : "\u0001",
"encoding" : "utf-8",
"fileType" : "text"
},
"name" : "hdfsreader"
},
"writer": {
"name": "hdfswriter",
"parameter": {
"path": "hdfs://nameservice1/user/hive/warehouse/pxb_test.db/prpjpayrefrec_oracle",
"defaultFS": "hdfs://nameservice1",
"column": [
{
"name":"city_id",
"type":"string"
},
{
"name":"city_name",
"type":"string"
}
],
"fileType": "text",
"maxFileSize": 10485760,
"nextCheckRows": 20000,
"fieldDelimiter": "\u0001",
"encoding": "utf-8",
"fileName": "dt=2023050813",
"writeMode": "overwrite",
"hadoopConfig": {
"dfs.ha.namenodes.nameservice1": "nn10,nn20",
"hadoop.user.name": "data_dev_pxb",
"fs.defaultFS": "hdfs://nameservice1",
"dfs.namenode.rpc-address.nameservice1.nn20": "twobigdata01:8020",
"dfs.client.failover.proxy.provider.nameservice1": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
"dfs.namenode.rpc-address.nameservice1.nn10": "twobigdata02:8020",
"dfs.nameservices": "nameservice1",
"fs.hdfs.impl.disable.cache": "true",
"fs.hdfs.impl": "org.apache.hadoop.hdfs.DistributedFileSystem",
"hadoop.security.authentication": "Kerberos",
"dfs.namenode.kerberos.principal": "hdfs/_HOST@TEST.COM",
"dfs.datanode.kerberos.principal": "hdfs/_HOST@TEST.COM",
"yarn.resourcemanager.principal": "rm/_HOST@TEST.COM",
"dfs.namenode.kerberos.internal.spnego.principal": "HTTP/_HOST@TEST.COM",
"hadoop.security.authorization": "true",
"dfs.namenode.keytab.file": "/app/chunjun/kerberos/data_dev_pxb.keytab",
"java.security.krb5.conf": "/app/chunjun/krb5/krb5_cdh2.conf",
"useLocalFile": "true",
"principalFile": "/app/chunjun/kerberos//data_dev_pxb.keytab",
"principal": "data_dev_pxb/twobigdata02@TEST.COM",
"dfs.client.use.datanode.hostname":"true",
"dfs.client.use.namenode.hostname":"true"
}
}
}
}
],
"setting": {
"speed": {
"channel": 1,
"bytes": 0
}
}
}
}
```
--------------我是分割线-------------
以上是我的json文件

### What you expected to happen

不知道在CDH之间同步数据,kerberos认证无法通过,是为什么,因为我是用stream插件,进行sync模式的时候,分别能通过chunjun读写上下游的CDH集群,但是两个CDH之间就是无法完成读写,总会报kerberos错误。(我确认kerberos账号及使用的keytab、krb5.conf文件没有问题,因为每一个cdh集群都能通过stream插件单独完成读写验证)

### How to reproduce

按照我写的json格式进行验证,就能复现问题

### Anything else

_No response_

### Version

master

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.