[Bug] Regardless of the client or cluster spark-mode, "spark.jar=xxx.jar" has been specified in spark conf, you cannot run "import XXXX" statement from the jar when executing spark scala core
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
### Regardless of the client or cluster spark-mode, "spark.jar=xxx.jar" has been specified in spark conf, you cannot run "import XXXX" statement from the jar when executing spark scala core.
#### 1. My kyuubi jdbc url
> private static String kyuubiJdbcUrl = "jdbc:hive2://xxxxx:9999/xxxxx;" +
> "?" +
> "kyuubi.engine.pool.name=USER-GROUP;" +
> "kyuubi.engine.pool.size=1;" +
> "kyuubi.session.engine.idle.timeout=PT15M;" +
> "#" +
> "spark.jars=s3://xxxx/*.jar;" +
> "spark.hadoop.mapreduce.input.pathFilter.class=org.apache.hudi.hadoop.HoodieROTablePathFilter;" +
> "spark.cassandra.connection.host=xxx;" +
> "spark.cassandra.connection.port=xxx;" +
> "spark.cassandra.auth.username=xxx;" +
> "spark.cassandra.auth.password=xxx;";
#### 2. My Spark code api
> import org.apache.spark.sql.{SaveMode, SparkSession}
> import com.datastax.spark.connector.cql.CassandraConnectorConf
> import org.apache.spark.sql.cassandra._
> spark.setCassandraConf(CassandraConnectorConf.KeepAliveMillisParam.option(10000))
> val writeDf = spark.read.parquet("xxxx")
> writeDf.printSchema()
> val cassandraMap = Map(table -> "xxxx", keyspace -> " xxxx")
> writeDf.write.format("org.apache.spark.sql.cassandra").options(cassandraMap).mode(SaveMode.Append).save()
#### 3. Error
> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error operating EXECUTE_STATEMENT: org.apache.kyuubi.KyuubiSQLException: Interpret error:
> import org.apache.spark.sql.{SaveMode, SparkSession}
> import com.datastax.spark.connector.cql.CassandraConnectorConf
> import org.apache.spark.sql.cassandra._
> spark.setCassandraConf(CassandraConnectorConf.KeepAliveMillisParam.option(10000))
> val writeDf = spark.read.parquet("s3:xxxx")
> writeDf.printSchema()
> val cassandraMap = Map("table" -> "xxxx", "keyspace" -> "xxxx")
> writeDf.write.format("org.apache.spark.sql.cassandra").options(cassandraMap).mode(SaveMode.Append).save()
>
> console:24: error: object datastax is not a member of package com
> import com.datastax.spark.connector.cql.CassandraConnectorConf
> ^
> console:25: error: object cassandra is not a member of package org.apache.spark.sql
> import org.apache.spark.sql.cassandra._
> ^
#### 4. Problem clue
> **We can find that it is possible to execute _’import org.apache.spark.sql.{savemode, sparksession}‘,_ but it is abnormal to execute 'import com.datastex.spark.connector.cql.cassandraconnectorconf'.**
>
> **In spark application UI, we found xxx.Jar has been introduced into _’spark.yarn.dist.jars'_ and _'spark.yarn.secondary.jars‘_, but cannot run "import XXXX" statement.**
>
> **We execute _’spark shell --jars xxx.jar'_, this code can be executed normally.**
`spark-shell --name my1 --conf "spark.cassandra.connection.host=xxxx" --conf "spark.cassandra.connection.port=9042" --conf "spark.cassandra.auth.username=xxxx" --conf "spark.cassandra.auth.password=xxx" --conf "spark.jars=s3:/xxxxx/*.jar"`
### Affects Version(s)
1.5.1
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
```yaml
spark.submit.deployMode=cluster/client
```
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Contributor guide
Research direction
Reproduce the failure using the JDBC URL with spark.jars and compare it with the provided spark-shell --jars command, checking both cluster and client deploy modes. Trace how Kyuubi passes configured jars to the Spark Scala engine; done means imports from the configured Cassandra jar work consistently, with a regression test if the engine test coverage is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cassandra, scala
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100