apache / apache/hudi

[SUPPORT] Compile Hudi 0.15 with Scala 2.13 and Spark 3.2

Open
#11,829 0 comments 0 reactions 0 assignees View on GitHub
area:build
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

Trying to compile Hudi 0.15 Scala 2.13

**To Reproduce**

```
Downloading from Maven Central: https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-spark3.2.x_2.13/0.15.0/hudi-spark3.2.x_2.13-0.15.0.jar
Downloading from cloudera-repo-releases: https://repository.cloudera.com/artifactory/public/org/apache/hudi/hudi-spark3.2.x_2.13/0.15.0/hudi-spark3.2.x_2.13-0.15.0.jar
Downloading from confluent: https://packages.confluent.io/maven/org/apache/hudi/hudi-spark3.2.x_2.13/0.15.0/hudi-spark3.2.x_2.13-0.15.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hudi 0.15.0:
[INFO]
[INFO] Hudi ............................................... SUCCESS [ 3.254 s]
[INFO] hudi-tests-common .................................. SUCCESS [ 5.256 s]
[INFO] hudi-io ............................................ SUCCESS [ 10.146 s]
[INFO] hudi-common ........................................ SUCCESS [ 20.071 s]
[INFO] hudi-hadoop-common ................................. SUCCESS [ 6.063 s]
[INFO] hudi-hadoop-mr ..................................... SUCCESS [ 4.787 s]
[INFO] hudi-sync-common ................................... SUCCESS [ 1.558 s]
[INFO] hudi-hive-sync ..................................... SUCCESS [ 6.594 s]
[INFO] hudi-aws ........................................... SUCCESS [ 2.667 s]
[INFO] hudi-timeline-service .............................. SUCCESS [ 1.951 s]
[INFO] hudi-client ........................................ SUCCESS [ 0.071 s]
[INFO] hudi-client-common ................................. SUCCESS [ 6.741 s]
[INFO] hudi-spark-client .................................. SUCCESS [ 33.252 s]
[INFO] hudi-spark-datasource .............................. SUCCESS [ 0.107 s]
[INFO] hudi-spark-common_2.13 ............................. SUCCESS [ 47.113 s]
[INFO] hudi-java-client ................................... SUCCESS [ 4.042 s]
[INFO] hudi-spark_2.13 .................................... FAILURE [ 2.327 s]
[INFO] hudi-gcp ........................................... SKIPPED
[INFO] hudi-utilities_2.13 ................................ SKIPPED
[INFO] hudi-utilities-bundle_2.13 ......................... SKIPPED
[INFO] hudi-cli ........................................... SKIPPED
[INFO] hudi-flink-client .................................. SKIPPED
[INFO] hudi-datahub-sync .................................. SKIPPED
[INFO] hudi-adb-sync ...................................... SKIPPED
[INFO] hudi-sync .......................................... SKIPPED
[INFO] hudi-hadoop-mr-bundle .............................. SKIPPED
[INFO] hudi-datahub-sync-bundle ........................... SKIPPED
[INFO] hudi-hive-sync-bundle .............................. SKIPPED
[INFO] hudi-aws-bundle .................................... SKIPPED
[INFO] hudi-gcp-bundle .................................... SKIPPED
[INFO] hudi-spark-bundle_2.13 ............................. SKIPPED
[INFO] hudi-presto-bundle ................................. SKIPPED
[INFO] hudi-utilities-slim-bundle_2.13 .................... SKIPPED
[INFO] hudi-timeline-server-bundle ........................ SKIPPED
[INFO] hudi-trino-bundle .................................. SKIPPED
[INFO] hudi-examples ...................................... SKIPPED
[INFO] hudi-examples-common ............................... SKIPPED
[INFO] hudi-examples-spark ................................ SKIPPED
[INFO] hudi-flink-datasource .............................. SKIPPED
[INFO] hudi-flink1.18.x ................................... SKIPPED
[INFO] hudi-flink ......................................... SKIPPED
[INFO] hudi-examples-flink ................................ SKIPPED
[INFO] hudi-examples-java ................................. SKIPPED
[INFO] hudi-flink1.14.x ................................... SKIPPED
[INFO] hudi-flink1.15.x ................................... SKIPPED
[INFO] hudi-flink1.16.x ................................... SKIPPED
[INFO] hudi-flink1.17.x ................................... SKIPPED
[INFO] hudi-kafka-connect ................................. SKIPPED
[INFO] hudi-flink1.18-bundle .............................. SKIPPED
[INFO] hudi-kafka-connect-bundle .......................... SKIPPED
[INFO] hudi-cli-bundle_2.13 ............................... SKIPPED
[INFO] hudi-hadoop-docker ................................. SKIPPED
[INFO] hudi-hadoop-base-docker ............................ SKIPPED
[INFO] hudi-hadoop-base-java11-docker ..................... SKIPPED
[INFO] hudi-hadoop-namenode-docker ........................ SKIPPED
[INFO] hudi-hadoop-datanode-docker ........................ SKIPPED
[INFO] hudi-hadoop-history-docker ......................... SKIPPED
[INFO] hudi-hadoop-hive-docker ............................ SKIPPED
[INFO] hudi-hadoop-sparkbase-docker ....................... SKIPPED
[INFO] hudi-hadoop-sparkmaster-docker ..................... SKIPPED
[INFO] hudi-hadoop-sparkworker-docker ..................... SKIPPED
[INFO] hudi-hadoop-sparkadhoc-docker ...................... SKIPPED
[INFO] hudi-hadoop-presto-docker .......................... SKIPPED
[INFO] hudi-hadoop-trinobase-docker ....................... SKIPPED
[INFO] hudi-hadoop-trinocoordinator-docker ................ SKIPPED
[INFO] hudi-hadoop-trinoworker-docker ..................... SKIPPED
[INFO] hudi-integ-test .................................... SKIPPED
[INFO] hudi-integ-test-bundle ............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:38 min
[INFO] Finished at: 2024-08-26T15:43:23-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hudi-spark_2.13: Could not resolve dependencies for project org.apache.hudi:hudi-spark_2.13:jar:0.15.0
[ERROR] dependency: org.apache.hudi:hudi-spark3.2.x_2.13:jar:0.15.0 (compile)
[ERROR] Could not find artifact org.apache.hudi:hudi-spark3.2.x_2.13:jar:0.15.0 in Maven Central (https://repo.maven.apache.org/maven2)
[ERROR] Could not find artifact org.apache.hudi:hudi-spark3.2.x_2.13:jar:0.15.0 in cloudera-repo-releases (https://repository.cloudera.com/artifactory/public/)
[ERROR] Could not find artifact org.apache.hudi:hudi-spark3.2.x_2.13:jar:0.15.0 in confluent (https://packages.confluent.io/maven/)
[ERROR] org.apache.hudi:hudi-spark3.2.x_2.13:jar:0.15.0 was not found in https://repo.maven.apache.org/maven2 during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced
[ERROR]
[ERROR] -> [Help 1]
```

Steps to reproduce the behavior:

1. `mvn clean package -Pintegration-tests -DskipTests -Dscala-2.13`
2.
3.
4.

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.15

* Spark version :

* Hive version :

* Hadoop version :

* Storage (HDFS/S3/GCS..) :

* Running on Docker? (yes/no) :

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running `mvn clean package -Pintegration-tests -DskipTests -Dscala-2.13` and inspect the `hudi-spark_2.13` module's dependency resolution. Focus on the missing `hudi-spark3.2.x_2.13:0.15.0` artifact reported by Maven. Done means the requested Hudi 0.15 Scala 2.13 build no longer fails on that unresolved dependency.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.