apache / apache/hudi

[SUPPORT] Error when I add non-bundle dependency

Open
#13,574 5 comments 0 reactions 0 assignees View on GitHub
area:build type:community-support
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

When I try to use dependency on hudi-spark3.5.x in SBT project, I get errors like
```
[error] https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.5.x_2.12/0.15.0/hudi-spark3.5.x_${scala.binary.version}-0.15.0.jar: not found: https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.5.x_2.12/0.15.0/hudi-spark3.5.x_${scala.binary.version}-0.15.0.jar
[error] https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark-common_2.12/0.15.0/hudi-spark-common_${scala.binary.version}-0.15.0.jar: not found: https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark-common_2.12/0.15.0/hudi-spark-common_${scala.binary.version}-0.15.0.jar
```

**To Reproduce**

Steps to reproduce the behavior:

1. Create a fresh minimal sbt project
2. Add build.sbt:
```
scalaVersion := "2.12.20"

name := "hudi-bundle"
version := "1.0"

lazy val root = (project in file(".")).
settings(libraryDependencies += "org.apache.hudi" %% "hudi-spark3.5.x" % "0.15.0")
```
3. Run `sbt compile`

**Expected behavior**

Project should be built

**Environment Description**

* Hudi version :
0.15.0 (1.0.2 has the same issue)

* Spark version :
3.5

* Running on Docker? (yes/no) :
No

**Stacktrace**

```
[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.5.x_2.12/0.15.0/hudi-spark3.5.x_${scala.binary.version}-0.15.0.jar: not found: https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.5.x_2.12/0.15.0/hudi-spark3.5.x_${scala.binary.version}-0.15.0.jar
[error] https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark-common_2.12/0.15.0/hudi-spark-common_${scala.binary.version}-0.15.0.jar: not found: https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark-common_2.12/0.15.0/hudi-spark-common_${scala.binary.version}-0.15.0.jar
[error]
[error] at lmcoursier.internal.shaded.coursier.Artifacts$.$anonfun$fetchArtifacts$9(Artifacts.scala:365)
[error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1(Task.scala:14)
[error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1$adapted(Task.scala:14)
[error] at lmcoursier.internal.shaded.coursier.util.Task$.wrap(Task.scala:82)
[error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14)
[error] at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
[error] at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:51)
[error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] at java.base/java.lang.Thread.run(Thread.java:1583)
[error] Caused by: lmcoursier.internal.shaded.coursier.cache.ArtifactError$NotFound: not found: https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.5.x_2.12/0.15.0/hudi-spark3.5.x_${scala.binary.version}-0.15.0.jar
[error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader$Blocking$.doDownload(Downloader.scala:239)
[error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader$Blocking$.$anonfun$remote$2(Downloader.scala:423)
[error] at lmcoursier.internal.shaded.coursier.cache.CacheLocks$.loop$1(CacheLocks.scala:94)
[error] at lmcoursier.internal.shaded.coursier.cache.CacheLocks$.withLockOr(CacheLocks.scala:117)
[error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader$Blocking$.$anonfun$remote$1(Downloader.scala:426)
[error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader$.$anonfun$downloading$2(Downloader.scala:776)
[error] at lmcoursier.internal.shaded.coursier.cache.CacheLocks$.withUrlLock(CacheLocks.scala:136)
[error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader$.$anonfun$downloading$1(Downloader.scala:776)
[error] at lmcoursier.internal.shaded.coursier.cache.internal.Retry.loop$1(Retry.scala:23)
[error] at lmcoursier.internal.shaded.coursier.cache.internal.Retry.retryOpt(Retry.scala:39)
[error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader$.coursier$cache$internal$Downloader$$downloading(Downloader.scala:812)
[error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader$Blocking$.remote(Downloader.scala:428)
[error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader.$anonfun$remote$6(Downloader.scala:546)
[error] at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[error] at scala.util.Success.$anonfun$map$1(Try.scala:255)
[error] at scala.util.Success.map(Try.scala:213)
[error] at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[error] at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42)
[error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] at java.base/java.lang.Thread.run(Thread.java:1583)
```

When I change dependency to `"org.apache.hudi" % "hudi-spark3.5.x_2.12" % "0.15.0"`, I get the same result with the same errors.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the minimal build.sbt and reproduce the failure using sbt compile. Compare the published Maven artifacts and metadata for hudi-spark3.5.x and hudi-spark-common with the literal ${scala.binary.version} paths in the error. Done means the dependency resolves from Maven Central and the sample project builds successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
build-system, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.