apache / apache/kyuubi

[Bug] One session add repeated jar, other session DML sqls will failed

Open
#3,781 0 comments 0 reactions 0 assignees View on GitHub
kind:bug priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the bug

If the user repeatedly adds the jars in one session , the dml sql from other session will fail. this is because AddJarCommand will add jar to common sparkContext, other session task failed when down dependent jars to container .

### Affects Version(s)

master/1.7

### Kyuubi Server Log Output

_No response_

### Kyuubi Engine Log Output

```logtalk
Caused by: org.apache.spark.SparkException: File ./xxx-0.0.1.jar exists and does not match contents of hdfs://user_zzz/yyy/udf/xxx-0.0.1.jar
at org.apache.spark.util.Utils$.copyFile(Utils.scala:685)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:542)
at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13(Executor.scala:963)
at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13$adapted(Executor.scala:955)
```

### Kyuubi Server Configurations

_No response_

### Kyuubi Engine Configurations

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [X] Yes. I can submit a PR independently to fix.
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Begin at the AddJarCommand entry point and reproduce repeated jar additions across two sessions, using the shown Spark executor dependency-fetch error as the symptom. Trace how the command reaches the shared SparkContext; done means DML from the other session no longer fails after repeated jar additions.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.