[Bug] Spark jar task failed to run
- Dominant language
- Java
- Stars
- 1.3k
- Forks
- 350
- PR merge metrics
- No merged PRs in 30d
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/DTStack/Taier/issues) and found no similar issues.
### What happened
When configuring the cluster components and running the Spark jar task, it was found that it could not run successfully
### What you expected to happen

### How to reproduce
I ran a spark pi task with parameters of 10 or 100, and the Application Master would link the parameters as hosts
Application application_1693541457708_0007 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1693541457708_0007_000001 exited with exitCode: 10
Failing this attempt.Diagnostics: [2023-09-01 15:56:35.718]Exception from container-launch.
Container id: container_e130_1693541457708_0007_01_000001
Exit code: 10
[2023-09-01 15:56:35.719]Container exited with a non-zero exit code 10. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
etrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Failed to connect to driver!
at org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkDriver(ApplicationMaster.scala:579)
at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:434)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:256)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:766)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:787)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
23/09/01 15:56:35 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: org.apache.spark.SparkException: Failed to connect to driver!)
23/09/01 15:56:35 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Uncaught exception: org.apache.spark.SparkException: Failed to connect to driver!)
23/09/01 15:56:35 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://lcc-ambari-server01:8020/user/admin/.sparkStaging/application_1693541457708_0007
23/09/01 15:56:35 INFO util.ShutdownHookManager: Shutdown hook called
[2023-09-01 15:56:35.719]Container exited with a non-zero exit code 10. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
etrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:33 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:34 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Failed to connect to driver at 10:0, retrying ...
23/09/01 15:56:35 ERROR yarn.ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Failed to connect to driver!
at org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkDriver(ApplicationMaster.scala:579)
at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:434)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:256)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:766)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:787)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
23/09/01 15:56:35 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: org.apache.spark.SparkException: Failed to connect to driver!)
23/09/01 15:56:35 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Uncaught exception: org.apache.spark.SparkException: Failed to connect to driver!)
23/09/01 15:56:35 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://lcc-ambari-server01:8020/user/admin/.sparkStaging/application_1693541457708_0007
23/09/01 15:56:35 INFO util.ShutdownHookManager: Shutdown hook called
For more detailed output, check the application tracking page: http://lcc-ambari-server01:8188/applicationhistory/app/application_1693541457708_0007 Then click on links to logs of each attempt.
. Failing the application.
### Anything else
_No response_
### Version
master
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.