[SUPPORT] Dependency hudi-spark-common_2.11 present in spark3.* 2.12 build
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
The released spark artifacts on maven (eg group: "org.apache.hudi", name: "hudi-spark3.2.x_2.12", version: "0.12.0") have an bad dependency on scala 2.11.
```
hudi-spark-common_${scala.binary.version}
```
but scala.binary.version is fixed to 2.11
https://github.com/apache/hudi/blob/master/pom.xml#L163
https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark3.2.x/pom.xml#L236
this leads to this error
```
org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.hudi.Spark3DefaultSource could not be instantiated
java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.hudi.Spark3DefaultSource could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:46)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.SparkAdapterSupport$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 114 more
```
unless you exclude the depency when importing :
```
exclude group: 'org.apache.hudi', module: 'hudi-spark-common_2.11'
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with pom.xml around line 163 and hudi-spark-datasource/hudi-spark3.2.x/pom.xml around line 236, then inspect the generated dependency for hudi-spark3.2.x_2.12. Verify that the released artifact no longer pulls hudi-spark-common_2.11 and that the reported Spark DataSource service-loading error is resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark
- Domain
- build-system, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100