[SUPPORT] Compile Hudi 0.15 with Scala 2.13 and Spark 3.2
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
Trying to compile Hudi 0.15 Scala 2.13
**To Reproduce**
```
Downloading from Maven Central: https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-spark3.2.x_2.13/0.15.0/hudi-spark3.2.x_2.13-0.15.0.jar
Downloading from cloudera-repo-releases: https://repository.cloudera.com/artifactory/public/org/apache/hudi/hudi-spark3.2.x_2.13/0.15.0/hudi-spark3.2.x_2.13-0.15.0.jar
Downloading from confluent: https://packages.confluent.io/maven/org/apache/hudi/hudi-spark3.2.x_2.13/0.15.0/hudi-spark3.2.x_2.13-0.15.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hudi 0.15.0:
[INFO]
[INFO] Hudi ............................................... SUCCESS [ 3.254 s]
[INFO] hudi-tests-common .................................. SUCCESS [ 5.256 s]
[INFO] hudi-io ............................................ SUCCESS [ 10.146 s]
[INFO] hudi-common ........................................ SUCCESS [ 20.071 s]
[INFO] hudi-hadoop-common ................................. SUCCESS [ 6.063 s]
[INFO] hudi-hadoop-mr ..................................... SUCCESS [ 4.787 s]
[INFO] hudi-sync-common ................................... SUCCESS [ 1.558 s]
[INFO] hudi-hive-sync ..................................... SUCCESS [ 6.594 s]
[INFO] hudi-aws ........................................... SUCCESS [ 2.667 s]
[INFO] hudi-timeline-service .............................. SUCCESS [ 1.951 s]
[INFO] hudi-client ........................................ SUCCESS [ 0.071 s]
[INFO] hudi-client-common ................................. SUCCESS [ 6.741 s]
[INFO] hudi-spark-client .................................. SUCCESS [ 33.252 s]
[INFO] hudi-spark-datasource .............................. SUCCESS [ 0.107 s]
[INFO] hudi-spark-common_2.13 ............................. SUCCESS [ 47.113 s]
[INFO] hudi-java-client ................................... SUCCESS [ 4.042 s]
[INFO] hudi-spark_2.13 .................................... FAILURE [ 2.327 s]
[INFO] hudi-gcp ........................................... SKIPPED
[INFO] hudi-utilities_2.13 ................................ SKIPPED
[INFO] hudi-utilities-bundle_2.13 ......................... SKIPPED
[INFO] hudi-cli ........................................... SKIPPED
[INFO] hudi-flink-client .................................. SKIPPED
[INFO] hudi-datahub-sync .................................. SKIPPED
[INFO] hudi-adb-sync ...................................... SKIPPED
[INFO] hudi-sync .......................................... SKIPPED
[INFO] hudi-hadoop-mr-bundle .............................. SKIPPED
[INFO] hudi-datahub-sync-bundle ........................... SKIPPED
[INFO] hudi-hive-sync-bundle .............................. SKIPPED
[INFO] hudi-aws-bundle .................................... SKIPPED
[INFO] hudi-gcp-bundle .................................... SKIPPED
[INFO] hudi-spark-bundle_2.13 ............................. SKIPPED
[INFO] hudi-presto-bundle ................................. SKIPPED
[INFO] hudi-utilities-slim-bundle_2.13 .................... SKIPPED
[INFO] hudi-timeline-server-bundle ........................ SKIPPED
[INFO] hudi-trino-bundle .................................. SKIPPED
[INFO] hudi-examples ...................................... SKIPPED
[INFO] hudi-examples-common ............................... SKIPPED
[INFO] hudi-examples-spark ................................ SKIPPED
[INFO] hudi-flink-datasource .............................. SKIPPED
[INFO] hudi-flink1.18.x ................................... SKIPPED
[INFO] hudi-flink ......................................... SKIPPED
[INFO] hudi-examples-flink ................................ SKIPPED
[INFO] hudi-examples-java ................................. SKIPPED
[INFO] hudi-flink1.14.x ................................... SKIPPED
[INFO] hudi-flink1.15.x ................................... SKIPPED
[INFO] hudi-flink1.16.x ................................... SKIPPED
[INFO] hudi-flink1.17.x ................................... SKIPPED
[INFO] hudi-kafka-connect ................................. SKIPPED
[INFO] hudi-flink1.18-bundle .............................. SKIPPED
[INFO] hudi-kafka-connect-bundle .......................... SKIPPED
[INFO] hudi-cli-bundle_2.13 ............................... SKIPPED
[INFO] hudi-hadoop-docker ................................. SKIPPED
[INFO] hudi-hadoop-base-docker ............................ SKIPPED
[INFO] hudi-hadoop-base-java11-docker ..................... SKIPPED
[INFO] hudi-hadoop-namenode-docker ........................ SKIPPED
[INFO] hudi-hadoop-datanode-docker ........................ SKIPPED
[INFO] hudi-hadoop-history-docker ......................... SKIPPED
[INFO] hudi-hadoop-hive-docker ............................ SKIPPED
[INFO] hudi-hadoop-sparkbase-docker ....................... SKIPPED
[INFO] hudi-hadoop-sparkmaster-docker ..................... SKIPPED
[INFO] hudi-hadoop-sparkworker-docker ..................... SKIPPED
[INFO] hudi-hadoop-sparkadhoc-docker ...................... SKIPPED
[INFO] hudi-hadoop-presto-docker .......................... SKIPPED
[INFO] hudi-hadoop-trinobase-docker ....................... SKIPPED
[INFO] hudi-hadoop-trinocoordinator-docker ................ SKIPPED
[INFO] hudi-hadoop-trinoworker-docker ..................... SKIPPED
[INFO] hudi-integ-test .................................... SKIPPED
[INFO] hudi-integ-test-bundle ............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:38 min
[INFO] Finished at: 2024-08-26T15:43:23-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hudi-spark_2.13: Could not resolve dependencies for project org.apache.hudi:hudi-spark_2.13:jar:0.15.0
[ERROR] dependency: org.apache.hudi:hudi-spark3.2.x_2.13:jar:0.15.0 (compile)
[ERROR] Could not find artifact org.apache.hudi:hudi-spark3.2.x_2.13:jar:0.15.0 in Maven Central (https://repo.maven.apache.org/maven2)
[ERROR] Could not find artifact org.apache.hudi:hudi-spark3.2.x_2.13:jar:0.15.0 in cloudera-repo-releases (https://repository.cloudera.com/artifactory/public/)
[ERROR] Could not find artifact org.apache.hudi:hudi-spark3.2.x_2.13:jar:0.15.0 in confluent (https://packages.confluent.io/maven/)
[ERROR] org.apache.hudi:hudi-spark3.2.x_2.13:jar:0.15.0 was not found in https://repo.maven.apache.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced
[ERROR]
[ERROR] -> [Help 1]
```
Steps to reproduce the behavior:
1. `mvn clean package -Pintegration-tests -DskipTests -Dscala-2.13`
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.15
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running `mvn clean package -Pintegration-tests -DskipTests -Dscala-2.13` and inspect the `hudi-spark_2.13` module's dependency resolution. Focus on the missing `hudi-spark3.2.x_2.13:0.15.0` artifact reported by Maven. Done means the requested Hudi 0.15 Scala 2.13 build no longer fails on that unresolved dependency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100