apache / apache/doris-spark-connector

[Feature] 优化 spark-doris-connector jar 大小

Open
#255 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
126
Avg merge
3d 10h
Merged PRs (30d)
4

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Description

建议优化一下spark-doris-connector jar 大小,当前jar 中包含了很多spark jars目录已经存在jar。
还包含calcite jar 是否必要?
image

### Use case

_No response_

### Related issues

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the spark-doris-connector JAR contents and identifying which Spark JARs and Calcite JARs are bundled. Review the connector's dependency and packaging configuration to determine which dependencies are already provided by Spark and whether Calcite is required. Done means producing a smaller connector JAR without unnecessary duplicate dependencies and verifying that it still works with its supported Spark environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
build-system, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.