apache / apache/kyuubi

[Bug] Regardless of the client or cluster spark-mode, "spark.jar=xxx.jar" has been specified in spark conf, you cannot run "import XXXX" statement from the jar when executing spark scala core

Open
#2,914 1 comment 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

### Regardless of the client or cluster spark-mode, "spark.jar=xxx.jar" has been specified in spark conf, you cannot run "import XXXX" statement from the jar when executing spark scala core.

#### 1. My kyuubi jdbc url

> private static String kyuubiJdbcUrl = "jdbc:hive2://xxxxx:9999/xxxxx;" +
> "?" +
> "kyuubi.engine.pool.name=USER-GROUP;" +
> "kyuubi.engine.pool.size=1;" +
> "kyuubi.session.engine.idle.timeout=PT15M;" +
> "#" +
> "spark.jars=s3://xxxx/*.jar;" +
> "spark.hadoop.mapreduce.input.pathFilter.class=org.apache.hudi.hadoop.HoodieROTablePathFilter;" +
> "spark.cassandra.connection.host=xxx;" +
> "spark.cassandra.connection.port=xxx;" +
> "spark.cassandra.auth.username=xxx;" +
> "spark.cassandra.auth.password=xxx;";

#### 2. My Spark code api

> import org.apache.spark.sql.{SaveMode, SparkSession}
> import com.datastax.spark.connector.cql.CassandraConnectorConf
> import org.apache.spark.sql.cassandra._
> spark.setCassandraConf(CassandraConnectorConf.KeepAliveMillisParam.option(10000))
> val writeDf = spark.read.parquet("xxxx")
> writeDf.printSchema()
> val cassandraMap = Map(table -> "xxxx", keyspace -> " xxxx")
> writeDf.write.format("org.apache.spark.sql.cassandra").options(cassandraMap).mode(SaveMode.Append).save()

#### 3. Error

> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: Error operating EXECUTE_STATEMENT: org.apache.kyuubi.KyuubiSQLException: Interpret error:
> import org.apache.spark.sql.{SaveMode, SparkSession}
> import com.datastax.spark.connector.cql.CassandraConnectorConf
> import org.apache.spark.sql.cassandra._
> spark.setCassandraConf(CassandraConnectorConf.KeepAliveMillisParam.option(10000))
> val writeDf = spark.read.parquet("s3:xxxx")
> writeDf.printSchema()
> val cassandraMap = Map("table" -> "xxxx", "keyspace" -> "xxxx")
> writeDf.write.format("org.apache.spark.sql.cassandra").options(cassandraMap).mode(SaveMode.Append).save()
>
> console:24: error: object datastax is not a member of package com
> import com.datastax.spark.connector.cql.CassandraConnectorConf
> ^
> console:25: error: object cassandra is not a member of package org.apache.spark.sql
> import org.apache.spark.sql.cassandra._
> ^

#### 4. Problem clue

> **We can find that it is possible to execute _’import org.apache.spark.sql.{savemode, sparksession}‘,_ but it is abnormal to execute 'import com.datastex.spark.connector.cql.cassandraconnectorconf'.**
>
> **In spark application UI, we found xxx.Jar has been introduced into _’spark.yarn.dist.jars'_ and _'spark.yarn.secondary.jars‘_, but cannot run "import XXXX" statement.**
>
> **We execute _’spark shell --jars xxx.jar'_, this code can be executed normally.**
`spark-shell --name my1 --conf "spark.cassandra.connection.host=xxxx" --conf "spark.cassandra.connection.port=9042" --conf "spark.cassandra.auth.username=xxxx" --conf "spark.cassandra.auth.password=xxx" --conf "spark.jars=s3:/xxxxx/*.jar"`

### Affects Version(s)

1.5.1

### Kyuubi Server Log Output

_No response_

### Kyuubi Engine Log Output

_No response_

### Kyuubi Server Configurations

_No response_

### Kyuubi Engine Configurations

```yaml
spark.submit.deployMode=cluster/client
```

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure using the JDBC URL with spark.jars and compare it with the provided spark-shell --jars command, checking both cluster and client deploy modes. Trace how Kyuubi passes configured jars to the Spark Scala engine; done means imports from the configured Cassandra jar work consistently, with a regression test if the engine test coverage is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, scala
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.