DTStack / DTStack/chunjun

The config of sftp

Open
#665 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
4.1k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

## General Question
below is myconfig
```
WITH (
'connector' = 'hive-x'
,'properties.hadoop.user.name' = 'hdfs'
,'properties.dfs.ha.namenodes.ns' = ''
,'properties.fs.defaultFS' = 'hdfs://xxx:8020'
,'properties.dfs.namenode.rpc-address.ns.nn2' = ''
,'properties.dfs.client.failover.proxy.provider.ns' = 'org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider'
,'properties.dfs.namenode.rpc-address.ns.nn1' = ''
,'properties.dfs.nameservices' = ''
,'properties.fs.hdfs.impl.disable.cache' = 'true'
,'properties.fs.hdfs.impl' = 'org.apache.hadoop.hdfs.DistributedFileSystem'
,'properties.hadoop.security.authorization' = 'true'
,'properties.hadoop.security.authentication' = 'Kerberos'
,'properties.dfs.namenode.kerberos.principal' = '"nn/_HOST@xxxx.COM'
,'properties.dfs.namenode.keytab.file' = '/etc/security/keytab/nn.service.keytab'
,'properties.java.security.krb5.conf' = '/etc/krb5.conf'
,'properties.principalFile' = '/etc/security/keytab/nn.service.keytab'
,'properties.remoteDir' = '/etc/security/keytab/'
**,'properties.sftp.host' = 'xxx'
,'properties.sftp.port' = '22'
,'properties.sftp.username' = 'xxx'
,'properties.sftp.password' = 'xxx'
--,'properties.useLocalFile' = 'true'**
,'default-fs' = 'hdfs://xxx:8020'
,'field-delimiter' = ','
,'encoding' = 'utf-8'
,'max-file-size' = '10485760'
,'next-check-rows' = '20000'
,'write-mode' = 'overwrite'
,'file-type' = 'parquet'

,'url' = 'jdbc:hive2://xxx:10000/ghydata;principal=hive/xxx04@GHYDATA.COM'
,'username' = 'xxx'
,'password' = ''
,'partition' = ''
,'partition-type' = 'DAY'
,'table-name' = 'sys_config'
```
error belows:
calling callable, 异常Msg:java.lang.IllegalArgumentException: The config of sftp is null
at com.dtstack.flinkx.security.SftpHandler.checkConfig(SftpHandler.java:110)
at com.dtstack.flinkx.security.SftpHandler.getInstance(SftpHandler.java:74)
at com.dtstack.flinkx.security.SftpHandler.lambda$getInstanceWithRetry$0(SftpHandler.java:67)
at com.dtstack.flinkx.util.RetryUtil$Retry.call(RetryUtil.java:141)
at com.dtstack.flinkx.util.RetryUtil$Retry.doRetry(RetryUtil.java:80)
at com.dtstack.flinkx.util.RetryUtil.executeWithRetry(RetryUtil.java:56)
at com.dtstack.flinkx.security.SftpHandler.getInstanceWithRetry(SftpHandler.java:67)
at com.dtstack.flinkx.security.KerberosUtil.loadFromSftp(KerberosUtil.java:281)
at com.dtstack.flinkx.security.KerberosUtil.loadFile(KerberosUtil.java:234)
at com.dtstack.flinkx.connector.hive.util.HiveDbUtil.getConnectionWithKerberos(HiveDbUtil.java:120)
at com.dtstack.flinkx.connector.hive.util.HiveDbUtil.getConnection(HiveDbUtil.java:85)
at com.dtstack.flinkx.connector.hive.util.HiveUtil.createHiveTableWithTableInfo(HiveUtil.java:83)
at com.dtstack.flinkx.connector.hive.sink.HiveOutputFormat.checkCreateTable(HiveOutputFormat.java:388)
at com.dtstack.flinkx.connector.hive.sink.HiveOutputFormat.primaryCreateTable(HiveOutputFormat.java:359)
at com.dtstack.flinkx.connector.hive.sink.HiveOutputFormat.openInternal(HiveOutputFormat.java:115)
at com.dtstack.flinkx.sink.format.BaseRichOutputFormat.open(BaseRichOutputFormat.java:225)
at com.dtstack.flinkx.connector.hive.sink.HiveOutputFormat.open(HiveOutputFormat.java:95)
at com.dtstack.flinkx.sink.DtOutputFormatSinkFunction.open(DtOutputFormatSinkFunction.java:95)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)

how do i config this in .sql file

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.