chunjun 1.12版本,配置了基于zk的ha,提交mysql_binlog sql任务报错
- Dominant language
- Java
- Stars
- 4.1k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
以下是报错日志:
2022-06-10 03:58:03.562 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost
2022-06-10 03:58:03.566 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2022-06-10 03:58:03.566 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.memory.process.size, 1600m
2022-06-10 03:58:03.566 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.process.size, 1728m
2022-06-10 03:58:03.567 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2022-06-10 03:58:03.567 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2022-06-10 03:58:03.567 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper
2022-06-10 03:58:03.567 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, hdfs:///flink/ha
2022-06-10 03:58:03.568 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, cdh-node-1:2181,cdh-node-2:2181,cdh-node-3:2181
2022-06-10 03:58:03.568 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.path.root, /flink
2022-06-10 03:58:03.568 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, /cluster_one
2022-06-10 03:58:03.568 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region
2022-06-10 03:58:03.568 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.bind-port, 8080-8090
log4j:ERROR Could not find value for key log4j.appender.logfile
log4j:ERROR Could not instantiate appender named "logfile".
2022-06-10 03:58:03.664 [main] WARN org.apache.flink.runtime.util.HadoopUtils - Could not find Hadoop configuration via any of the supported methods (Flink configuration, environment variables).
2022-06-10 03:58:04,111 - 0 WARN [main] org.apache.hadoop.util.NativeCodeLoader:Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-06-10 03:58:04.291 [main] INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to app (auth:SIMPLE)
2022-06-10 03:58:04.336 [main] INFO org.apache.flink.runtime.security.modules.JaasModule - Jaas file will be created as /tmp/jaas-4266574835055373226.conf.
2022-06-10 03:58:04,554 - 443 INFO [main] org.apache.hadoop.yarn.client.RMProxy:Connecting to ResourceManager at /0.0.0.0:8032
2022-06-10 03:58:04.852 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2022-06-10 03:58:05.121 [main] WARN org.apache.flink.yarn.YarnClusterDescriptor - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
2022-06-10 03:58:05.318 [main] INFO com.dtstack.chunjun.Main - ------------program params-------------------------
2022-06-10 03:58:05.319 [main] INFO com.dtstack.chunjun.Main - -flinkLibDir
2022-06-10 03:58:05.319 [main] INFO com.dtstack.chunjun.Main - /home/ap/app/flink/flink-1.12.7/lib
2022-06-10 03:58:05.319 [main] INFO com.dtstack.chunjun.Main - -p
2022-06-10 03:58:05.319 [main] INFO com.dtstack.chunjun.Main -
2022-06-10 03:58:05.319 [main] INFO com.dtstack.chunjun.Main - -job
2022-06-10 03:58:05.319 [main] INFO com.dtstack.chunjun.Main - CREATE+TABLE+source%0A%28%0A++++id+++varchar%2C%0A++++name+++++++varchar%2C%0A++++version+float%0A%29+WITH+%28%0A++++++%27connector%27+%3D+%27binlog-x%27%0A++++++%2C%27username%27+%3D+%27root%27%0A++++++%2C%27password%27+%3D+%27123456%27%0A++++++%2C%27cat%27+%3D+%27insert%2Cdelete%2Cupdate%27%0A++++++%2C%27url%27+%3D+%27jdbc%3Amysql%3A%2F%2F192.168.8.127%3A33061%2Ftest_db%3FuseSSL%3Dfalse%27%0A++++++%2C%27host%27+%3D+%27192.168.8.127%27%0A++++++%2C%27port%27+%3D+%2733061%27%0A--+++%2C%27journal-name%27+%3D+%27mysql-bin.000001%27%0A++++++%2C%27table%27+%3D+%27test_db.record%27%0A++++++%2C%27timestamp-format.standard%27+%3D+%27SQL%27%0A++++++%29%3B%0A%0ACREATE+TABLE+sink%0A%28%0A++++id%09%09varchar%2C%0A++++name%09varchar%2C%0A++++version%09float%2C%0A++++PRIMARY+KEY+%28id%2C+name%2C+version%29+NOT+ENFORCED%0A%29+WITH+%28%0A--++++++%27connector%27+%3D+%27stream-x%27%0A%09%27connector%27+%3D+%27upsert-kafka-x%27%2C%0A%09%27topic%27+%3D+%27mysql_cdc%27%2C%0A%09%27properties.bootstrap.servers%27+%3D+%27cdh-node-2%3A9092%2Ccdh-node-3%3A9092%27%2C%0A%09%27key.format%27+%3D+%27json%27%2C%0A%09%27value.format%27+%3D+%27json%27%2C%0A%09%27value.fields-include%27+%3D+%27ALL%27%2C%0A%09%27sink.parallelism%27+%3D+%272%27%0A%09--%2C%27json.timestamp-format.standard%27+%3D+%27SQL%27%0A++++++%29%3B%0A%0A%0ACREATE+TABLE+print%0A%28%0A++++id++++++++++varchar%2C%0A++++name++++++++varchar%2C%0A++++version+++++float%2C%0A++++PRIMARY+KEY+%28id%2C+name%2C+version%29+NOT+ENFORCED%0A%29+WITH+%28%0A++++++%27connector%27+%3D+%27stream-x%27%0A++++++++--%2C%27json.timestamp-format.standard%27+%3D+%27SQL%27%0A++++++%29%3B%0A%0A%0Ainsert+into+print%0Aselect+*%0Afrom+source+u%3B%0A
2022-06-10 03:58:05.320 [main] INFO com.dtstack.chunjun.Main - -jobName
2022-06-10 03:58:05.320 [main] INFO com.dtstack.chunjun.Main - mysql_binlog
2022-06-10 03:58:05.320 [main] INFO com.dtstack.chunjun.Main - -flinkxDistDir
2022-06-10 03:58:05.320 [main] INFO com.dtstack.chunjun.Main - /home/ap/app/flink/chunjun-dist
2022-06-10 03:58:05.320 [main] INFO com.dtstack.chunjun.Main - -chunjunDistDir
2022-06-10 03:58:05.320 [main] INFO com.dtstack.chunjun.Main - /home/ap/app/flink/chunjun-dist
2022-06-10 03:58:05.320 [main] INFO com.dtstack.chunjun.Main - -jobType
2022-06-10 03:58:05.320 [main] INFO com.dtstack.chunjun.Main - sql
2022-06-10 03:58:05.320 [main] INFO com.dtstack.chunjun.Main - -hadoopConfDir
2022-06-10 03:58:05.320 [main] INFO com.dtstack.chunjun.Main - /home/ap/app/flink/hadoop-conf
2022-06-10 03:58:05.320 [main] INFO com.dtstack.chunjun.Main - -confProp
2022-06-10 03:58:05.321 [main] INFO com.dtstack.chunjun.Main - {"flink.checkpoint.interval":60000,"yarn.application.queue":"root.default"}
2022-06-10 03:58:05.321 [main] INFO com.dtstack.chunjun.Main - -pluginLoadMode
2022-06-10 03:58:05.321 [main] INFO com.dtstack.chunjun.Main - shipfile
2022-06-10 03:58:05.321 [main] INFO com.dtstack.chunjun.Main - -mode
2022-06-10 03:58:05.321 [main] INFO com.dtstack.chunjun.Main - yarn-per-job
2022-06-10 03:58:05.321 [main] INFO com.dtstack.chunjun.Main - -flinkConfDir
2022-06-10 03:58:05.321 [main] INFO com.dtstack.chunjun.Main - /home/ap/app/flink/flink-1.12.7/conf
2022-06-10 03:58:05.321 [main] INFO com.dtstack.chunjun.Main - -------------------------------------------
2022-06-10 03:58:05.328 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost
2022-06-10 03:58:05.329 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2022-06-10 03:58:05.329 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.memory.process.size, 1600m
2022-06-10 03:58:05.329 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.process.size, 1728m
2022-06-10 03:58:05.329 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2022-06-10 03:58:05.330 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2022-06-10 03:58:05.330 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability, zookeeper
2022-06-10 03:58:05.330 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.storageDir, hdfs:///flink/ha
2022-06-10 03:58:05.331 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.quorum, cdh-node-1:2181,cdh-node-2:2181,cdh-node-3:2181
2022-06-10 03:58:05.331 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.zookeeper.path.root, /flink
2022-06-10 03:58:05.331 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, /cluster_one
2022-06-10 03:58:05.331 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.execution.failover-strategy, region
2022-06-10 03:58:05.332 [main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.bind-port, 8080-8090
2022-06-10 03:58:07.227 [main] INFO com.dtstack.chunjun.Main - Register to table configuration:{table.dynamic-table-options.enabled=true, pipeline.name=mysql_binlog}
2022-06-10 03:58:07.229 [main] WARN com.dtstack.chunjun.options.Options - Option 'flinkxDistDir' is deprecated, please replace with 'chunjunDistDir'.
2022-06-10 03:58:07.230 [main] WARN com.dtstack.chunjun.options.Options - Option 'flinkxDistDir' is deprecated, please replace with 'chunjunDistDir'.
2022-06-10 03:58:07.268 [main] INFO com.dtstack.chunjun.util.PluginUtil - Flinkx executionMode: yarn-per-job
2022-06-10 03:58:07.268 [main] INFO com.dtstack.chunjun.util.PluginUtil - Flinkx reset pipeline.jars: [file:/home/ap/app/flink/chunjun-dist/dirty-data-collector/log/chunjun-dirty-log-master.jar]
2022-06-10 03:58:07.917 [main] INFO com.dtstack.chunjun.util.PluginUtil - Flinkx executionMode: yarn-per-job
2022-06-10 03:58:07.918 [main] INFO com.dtstack.chunjun.util.PluginUtil - Flinkx reset pipeline.jars: [file:/home/ap/app/flink/chunjun-dist/dirty-data-collector/log/chunjun-dirty-log-master.jar, file:/home/ap/app/flink/chunjun-dist/connector/binlog/chunjun-connector-binlog-master.jar]
2022-06-10 03:58:08.721 [main] INFO com.dtstack.chunjun.util.PluginUtil - Flinkx executionMode: yarn-per-job
2022-06-10 03:58:08.721 [main] INFO com.dtstack.chunjun.util.PluginUtil - Flinkx reset pipeline.jars: [file:/home/ap/app/flink/chunjun-dist/dirty-data-collector/log/chunjun-dirty-log-master.jar, file:/home/ap/app/flink/chunjun-dist/connector/binlog/chunjun-connector-binlog-master.jar, file:/home/ap/app/flink/chunjun-dist/connector/stream/chunjun-connector-stream-master.jar]
2022-06-10 03:58:10.992 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024, numberTaskManagers=1, slotsPerTaskManager=1, priority=0}
2022-06-10 03:58:10.997 [main] WARN org.apache.flink.core.plugin.PluginConfig - The plugins directory [plugins] does not exist.
2022-06-10 03:58:11.134 [main] WARN org.apache.flink.yarn.YarnClusterDescriptor - The file system scheme is 'file'. This indicates that the specified Hadoop configuration path is wrong and the system is using the default Hadoop configuration values.The Flink YARN client needs to store its files in a distributed file system
2022-06-10 03:58:11.436 [main] WARN org.apache.flink.core.plugin.PluginConfig - The plugins directory [plugins] does not exist.
2022-06-10 03:58:11.746 [main] INFO o.apache.flink.runtime.util.config.memory.ProcessMemoryUtils - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-06-10 03:58:11.759 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - Submitting application master application_1654566646340_0003
2022-06-10 03:58:11,826 - 7715 INFO [main] org.apache.hadoop.yarn.client.api.impl.YarnClientImpl:Submitted application application_1654566646340_0003
2022-06-10 03:58:11.826 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - Waiting for the cluster to be allocated
2022-06-10 03:58:11.828 [main] INFO org.apache.flink.yarn.YarnClusterDescriptor - Deploying cluster, current state ACCEPTED
Exception in thread "main" org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster.
at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:491)
at com.dtstack.chunjun.client.yarn.YarnPerJobClusterClientHelper.submit(YarnPerJobClusterClientHelper.java:97)
at com.dtstack.chunjun.client.Launcher.main(Launcher.java:119)
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment.
Diagnostics from YARN: Application application_1654566646340_0003 failed 2 times in previous 10000 milliseconds due to AM Container for appattempt_1654566646340_0003_000002 exited with exitCode: -1000
Failing this attempt.Diagnostics: [2022-06-10 03:58:16.565]File file:/home/ap/app/.flink/application_1654566646340_0003/flink-shaded-zookeeper-3.4.14.jar does not exist
java.io.FileNotFoundException: File file:/home/ap/app/.flink/application_1654566646340_0003/flink-shaded-zookeeper-3.4.14.jar does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
at org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:269)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:411)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:411)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:242)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:235)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.call(ContainerLocalizer.java:223)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
For more detailed output, check the application tracking page: http://cdh-node-1:8088/cluster/app/application_1654566646340_0003 Then click on links to logs of each attempt.
. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1654566646340_0003
at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1251)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:634)
at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:484)
... 2 more
我执行的脚本:
java -cp /home/ap/app/flink/chunjun-dist/connector/chunjun-clients-master.jar com.dtstack.chunjun.client.Launcher \
-mode yarn-per-job \
-jobType sql \
-jobName mysql_binlog \
-flinkConfDir /home/ap/app/flink/flink-1.12.7/conf \
-hadoopConfDir /home/ap/app/flink/hadoop-conf \
-flinkLibDir /home/ap/app/flink/flink-1.12.7/lib \
-confProp "{\"flink.checkpoint.interval\":60000,\"yarn.application.queue\":\"root.default\"}" \
-job /home/ap/app/flink/chunjun-dist/sql/binlog/mysql_binlog.sql \
-chunjunDistDir /home/ap/app/flink/chunjun-dist \
flink-conf.yaml关于zk ha的配置:
high-availability: zookeeper
high-availability.storageDir: hdfs:///flink/ha
high-availability.zookeeper.quorum: cdh-node-1:2181,cdh-node-2:2181,cdh-node-3:2181
high-availability.zookeeper.path.root: /flink
high-availability.cluster-id: /cluster_one
执行的sql内容:
CREATE TABLE source
(
id varchar,
name varchar,
version float
) WITH (
'connector' = 'binlog-x'
,'username' = 'root'
,'password' = '123456'
,'cat' = 'insert,delete,update'
,'url' = 'jdbc:mysql://192.168.8.1127:33061/test_db?useSSL=false'
,'host' = '192.168.8.1'
,'port' = '33061'
-- ,'journal-name' = 'mysql-bin.000001'
,'table' = 'test_db.record'
,'timestamp-format.standard' = 'SQL'
);
CREATE TABLE print
(
id varchar,
name varchar,
version float,
PRIMARY KEY (id, name, version) NOT ENFORCED
) WITH (
'connector' = 'stream-x'
--,'json.timestamp-format.standard' = 'SQL'
);
insert into print
select *
from source u;
Contributor guide
Assessment
This issue has not been assessed yet.