DTStack / DTStack/Taier

Spark V3.2 Deploy Mode client 时 执行SPARK SQL 任务报错

Open
#1,186 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Java
Stars
1.3k
Forks
350
PR merge metrics
No merged PRs in 30d

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/DTStack/Taier/issues) and found no similar question.

- [X] I had searched my question on the internet but i didn't get any help.

- [X] I had read the documentation: [Taier doc](https://dtstack.github.io/Taier) but it didn't help me.

### Description

Spark v3.2 配置项如下 :

spark.submit.deployMode : client

执行如下 Spark SQL 任务:

-- name TestSpark
-- type SparkSQL
-- author mumu@h2.com
-- create time 2024-07-18 10:27:21
-- desc test

-- show databases;

--- show tables in tmp

select count(1) from tmp.nono_translate_id_log

-----------------------------------------------------
发生如下异常:

[10:34:22] 第1条任务开始执行
[10:34:24] 等待运行.....
[10:34:26] 等待运行.....
[10:34:27] 等待运行.....
[10:34:29] 等待运行.....
[10:34:29] 运行失败
[10:34:31] ====================基本日志====================
2024-07-18 10:34:23:submit job is success
====================appLogs====================
Uncaught exception: org.apache.spark.SparkException: Invalid Spark URL: spark://YarnScheduler@%7B%22appName%22%3A%22run_sql_task_1721270063116%22%2C%22userName%22%3A%22mumu%40h2.com%22%2C%22sql%22%3A%22UEsDBBQACAgIAEtU8lgAAAAAAAAAAAAAAAABAAAAMC2NQQ7CMAwEv%2BJjywHRJqQHHmOFxEVITlxi%5Cn5%2F%2BllNNqRxpNV4JMa%2BxsD0iNohFYfDKBElMy1A%2BjUdnwR9FPbnFL8Lcp3F2Y%2FQxq0ihDVJCWjjlF%5CnuMDapMDwv0l6tWEaT2plu1apgtZiVf5W8Z2R5TUerR1QSwcI0%2F4SoHMAAACVAAAA%5Cn%22%2C%22sparkSessionConf%22%3A%7B%22hive.default.fileformat%22%3A%22orc%22%7D%7D:0
at org.apache.spark.rpc.RpcEndpointAddress$.apply(RpcEndpointAddress.scala:66)
at org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:140)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:110)
at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:553)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:267)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:934)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:933)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:933)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:965)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)

**在 spark.submit.deployMode : cluster 模式下,是可以正常运行的 。**

求指点, 谢谢

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.