apache / apache/hudi

[SUPPORT] Dependency hudi-spark-common_2.11 present in spark3.* 2.12 build

Open
#6,767 7 comments 0 reactions 0 assignees View on GitHub
dependencies priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

The released spark artifacts on maven (eg group: "org.apache.hudi", name: "hudi-spark3.2.x_2.12", version: "0.12.0") have an bad dependency on scala 2.11.

```
hudi-spark-common_${scala.binary.version}
```
but scala.binary.version is fixed to 2.11

https://github.com/apache/hudi/blob/master/pom.xml#L163
https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark3.2.x/pom.xml#L236

this leads to this error
```
org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.hudi.Spark3DefaultSource could not be instantiated
java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.hudi.Spark3DefaultSource could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:46)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.SparkAdapterSupport$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 114 more
```

unless you exclude the depency when importing :
```
exclude group: 'org.apache.hudi', module: 'hudi-spark-common_2.11'
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with pom.xml around line 163 and hudi-spark-datasource/hudi-spark3.2.x/pom.xml around line 236, then inspect the generated dependency for hudi-spark3.2.x_2.12. Verify that the released artifact no longer pulls hudi-spark-common_2.11 and that the reported Spark DataSource service-loading error is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark
Domain
build-system, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.