Provider org.apache.hadoop.fs.sftp.SFTPFileSystem not found
- Dominant language
- Java
- Stars
- 4.1k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
**版本**:v.1.12.1
以yarn-per-job运行,
```
bin/flinkx -mode yarn-per-job -jobType sync -job flinkx-examples/json/stream/stream.json -flinkxDistDir /opt/chunjun/flinkx-dist/ -flinkConfDir /etc/flink/conf -hadoopConfDir /etc/hadoop/conf -flinkLibDir /opt/cloudera/parcels/FLINK/lib/flink/lib
```
报错如下:
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/chunjun/flinkx-dist/lib/flinkx-clients-1.12_release.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/chunjun/flinkx-dist/lib/flinkx-connector-inceptor-1.12_release.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/chunjun/flinkx-dist/lib/flinkx-restore-mysql-1.12_release.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
15:31:52,033 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
15:31:52,033 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
15:31:52,033 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/opt/chunjun/flinkx-dist/lib/flinkx-clients-1.12_release.jar!/logback.xml]
15:31:52,034 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
15:31:52,034 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/opt/chunjun/flinkx-dist/lib/flinkx-dirty-log-1.12_release.jar!/logback.xml]
15:31:52,034 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/opt/chunjun/flinkx-dist/lib/flinkx-clients-1.12_release.jar!/logback.xml]
15:31:52,050 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@2a2d45ba - URL [jar:file:/opt/chunjun/flinkx-dist/lib/flinkx-clients-1.12_release.jar!/logback.xml] is not of type file
15:31:52,154 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
15:31:52,160 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Will scan for changes in [jar:file:/opt/chunjun/flinkx-dist/lib/flinkx-clients-1.12_release.jar!/logback.xml]
15:31:52,160 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Setting ReconfigureOnChangeTask scanning period to 1 minutes
15:31:52,169 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
15:31:52,171 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [console]
15:31:52,178 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
15:31:52,220 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
15:31:52,220 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [console] to Logger[ROOT]
15:31:52,221 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.dtstack.flinkx] to TRACE
15:31:52,221 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [com.dtstack.flinkx] to false
15:31:52,221 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [console] to Logger[com.dtstack.flinkx]
15:31:52,221 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
15:31:52,222 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@2a5ca609 - Registering current configuration as safe fallback point
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
2022-03-07 15:31:52.572 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost
2022-03-07 15:31:52.575 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2022-03-07 15:31:52.575 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.memory.process.size, 1600m
2022-03-07 15:31:52.575 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.process.size, 1728m
2022-03-07 15:31:52.576 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 2
2022-03-07 15:31:52.576 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2022-03-07 15:31:52.576 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region
2022-03-07 15:31:52.576 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: classloader.check-leaked-classloader, false
2022-03-07 15:31:52.847 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-03-07 15:31:52.993 [main] INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to root (auth:SIMPLE)
2022-03-07 15:31:53.022 [main] INFO org.apache.flink.runtime.security.modules.JaasModule - Jaas file will be created as /tmp/jaas-580198448047640401.conf.
2022-03-07 15:31:53.147 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at test-4-178/10.30.4.178:8032
2022-03-07 15:31:53.264 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
log4j:ERROR Could not find value for key log4j.appender.logfile
log4j:ERROR Could not instantiate appender named "logfile".
2022-03-07 15:31:53.439 [main] WARN org.apache.flink.yarn.YarnClusterDescriptor - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2022-03-07 15:31:53.544 [main] INFO com.dtstack.flinkx.Main - ------------program params-------------------------
2022-03-07 15:31:53.544 [main] INFO com.dtstack.flinkx.Main - -flinkLibDir
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - /opt/cloudera/parcels/FLINK/lib/flink/lib
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - -p
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main -
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - -job
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - {
"job": {
"content": [
{
"reader": {
"parameter": {
"column": [
{
"name": "id",
"type": "id"
},
{
"name": "name",
"type": "string"
},
{
"name": "content",
"type": "string"
}
],
"sliceRecordCount": ["30"],
"permitsPerSecond": 1
},
"table": {
"tableName": "sourceTable"
},
"name": "streamreader"
},
"writer": {
"parameter": {
"column": [
{
"name": "id",
"type": "id"
},
{
"name": "name",
"type": "string"
},
{
"name": "content",
"type": "timestamp"
}
],
"print": true
},
"table": {
"tableName": "sinkTable"
},
"name": "streamwriter"
},
"transformer": {
"transformSql": "select id,name, NOW() from sourceTable where CHAR_LENGTH(name) < 50 and CHAR_LENGTH(content) < 50"
}
}
],
"setting": {
"errorLimit": {
"record": 100
},
"speed": {
"bytes": 0,
"channel": 1,
"readerChannel": 1,
"writerChannel": 1
}
}
}
}
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - -jobName
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - Flink_Job
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - -flinkxDistDir
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - /opt/chunjun/flinkx-dist/
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - -jobType
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - sync
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - -hadoopConfDir
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - /etc/hadoop/conf
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - -confProp
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - {}
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - -pluginLoadMode
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - shipfile
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - -mode
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - yarn-per-job
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - -flinkConfDir
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - /etc/flink/conf
2022-03-07 15:31:53.545 [main] INFO com.dtstack.flinkx.Main - -------------------------------------------
2022-03-07 15:31:53.548 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost
2022-03-07 15:31:53.548 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2022-03-07 15:31:53.548 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.memory.process.size, 1600m
2022-03-07 15:31:53.548 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.process.size, 1728m
2022-03-07 15:31:53.548 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 2
2022-03-07 15:31:53.548 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2022-03-07 15:31:53.548 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region
2022-03-07 15:31:53.549 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: classloader.check-leaked-classloader, false
2022-03-07 15:31:54.855 [main] INFO com.dtstack.flinkx.classloader.ClassLoaderManager - jarUrl:file:/opt/chunjun/flinkx-dist/connector/stream/flinkx-connector-stream-1.12_release.jar_file:/opt/chunjun/flinkx-dist/dirty-data-collector/log/flinkx-dirty-log-1.12_release.jar_file:/opt/chunjun/flinkx-dist/flinkx-core-1.12_release.jar_file:/opt/chunjun/flinkx-dist/metrics/prometheus/flinkx-metrics-prometheus-1.12_release.jar create ClassLoad successful...
2022-03-07 15:31:57.278 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024, numberTaskManagers=1, slotsPerTaskManager=1, priority=0}
2022-03-07 15:31:57.283 [main] WARN org.apache.flink.core.plugin.PluginConfig - The plugins directory [plugins] does not exist.
2022-03-07 15:31:57.415 [main] WARN org.apache.hadoop.fs.FileSystem - Cannot load filesystem
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.sftp.SFTPFileSystem not found
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2629)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2648)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2665)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2701)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2683)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:372)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:171)
at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:811)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:634)
at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:484)
at com.dtstack.flinkx.client.yarn.YarnPerJobClusterClientHelper.submit(YarnPerJobClusterClientHelper.java:97)
at com.dtstack.flinkx.client.Launcher.main(Launcher.java:126)
2022-03-07 15:31:59.708 [main] WARN org.apache.flink.core.plugin.PluginConfig - The plugins directory [plugins] does not exist.
2022-03-07 15:32:00.898 [main] INFO o.apache.flink.runtime.util.config.memory.ProcessMemoryUtils - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-03-07 15:32:00.910 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - Submitting application master application_1645686071960_0027
2022-03-07 15:32:00.974 [main] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1645686071960_0027
2022-03-07 15:32:00.974 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - Waiting for the cluster to be allocated
2022-03-07 15:32:00.976 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - Deploying cluster, current state ACCEPTED
2022-03-07 15:32:09.516 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - YARN application has been deployed successfully.
2022-03-07 15:32:09.517 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - The Flink YARN session cluster has been started in detached mode. In order to stop Flink gracefully, use the following command:
$ echo "stop" | ./bin/yarn-session.sh -id application_1645686071960_0027
If this should not be possible, then you can also kill Flink via YARN's web interface or via:
$ yarn application -kill application_1645686071960_0027
Note that killing Flink might not clean up all job artifacts and temporary files.
2022-03-07 15:32:09.518 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - Found Web Interface test-4-177:21808 of application 'application_1645686071960_0027'.
2022-03-07 15:32:09.623 [main] INFO com.dtstack.flinkx.client.yarn.YarnPerJobClusterClientHelper - deploy per_job with appId: application_1645686071960_0027}, jobId: 57456057c114b41094a78282e2cb4276
```
Contributor guide
Assessment
This issue has not been assessed yet.