apache / apache/hudi

Migrate Hudi to proper Incremental Build System (Bazel)

Open
#14,935 0 comments 0 reactions 0 assignees View on GitHub
area:dev-experience from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Even though Maven claims that it supports incremental builds, that is not what i personally observed in a day to day operations: even single file change trickles down into *whole project* being rebuilt which is currently taking about {*}6.5 mins{*}.

This is a substantial productivity drain, so migration to more modern Build System (like Bazel), while not easy, should bring the fruits quite fast, paying itself off well over by just how much time we would be saving on not waiting for these builds to complete.

 
{code:java}
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hudi 0.11.0-SNAPSHOT:
[INFO]
[INFO] Hudi ............................................... SUCCESS [  5.150 s]
[INFO] hudi-common ........................................ SUCCESS [ 18.957 s]
[INFO] hudi-aws ........................................... SUCCESS [  4.052 s]
[INFO] hudi-timeline-service .............................. SUCCESS [  5.437 s]
[INFO] hudi-client ........................................ SUCCESS [  0.291 s]
[INFO] hudi-client-common ................................. SUCCESS [ 13.902 s]
[INFO] hudi-hadoop-mr ..................................... SUCCESS [  5.227 s]
[INFO] hudi-spark-client .................................. SUCCESS [ 23.578 s]
[INFO] hudi-sync-common ................................... SUCCESS [  1.630 s]
[INFO] hudi-hive-sync ..................................... SUCCESS [ 10.545 s]
[INFO] hudi-spark-datasource .............................. SUCCESS [  0.090 s]
[INFO] hudi-spark-common_2.12 ............................. SUCCESS [ 17.497 s]
[INFO] hudi-spark3_2.12 ................................... SUCCESS [ 10.558 s]
[INFO] hudi-spark_2.12 .................................... SUCCESS [ 46.709 s]
[INFO] hudi-utilities_2.12 ................................ SUCCESS [  9.061 s]
[INFO] hudi-utilities-bundle_2.12 ......................... SUCCESS [ 18.046 s]
[INFO] hudi-cli ........................................... SUCCESS [ 16.890 s]
[INFO] hudi-java-client ................................... SUCCESS [  3.887 s]
[INFO] hudi-flink-client .................................. SUCCESS [ 10.512 s]
[INFO] hudi-spark2_2.12 ................................... SUCCESS [ 15.522 s]
[INFO] hudi-dla-sync ...................................... SUCCESS [  2.017 s]
[INFO] hudi-sync .......................................... SUCCESS [  0.075 s]
[INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [  6.188 s]
[INFO] hudi-hive-sync-bundle .............................. SUCCESS [  1.783 s]
[INFO] hudi-spark3-bundle_2.12 ............................ SUCCESS [ 17.597 s]
[INFO] hudi-presto-bundle ................................. SUCCESS [  4.982 s]
[INFO] hudi-timeline-server-bundle ........................ SUCCESS [  6.391 s]
[INFO] hudi-hadoop-docker ................................. SUCCESS [  2.110 s]
[INFO] hudi-hadoop-base-docker ............................ SUCCESS [  1.404 s]
[INFO] hudi-hadoop-namenode-docker ........................ SUCCESS [  1.124 s]
[INFO] hudi-hadoop-datanode-docker ........................ SUCCESS [  1.090 s]
[INFO] hudi-hadoop-history-docker ......................... SUCCESS [  1.098 s]
[INFO] hudi-hadoop-hive-docker ............................ SUCCESS [  2.019 s]
[INFO] hudi-hadoop-sparkbase-docker ....................... SUCCESS [  0.993 s]
[INFO] hudi-hadoop-sparkmaster-docker ..................... SUCCESS [  1.106 s]
[INFO] hudi-hadoop-sparkworker-docker ..................... SUCCESS [  1.063 s]
[INFO] hudi-hadoop-sparkadhoc-docker ...................... SUCCESS [  1.031 s]
[INFO] hudi-hadoop-presto-docker .......................... SUCCESS [  1.103 s]
[INFO] hudi-integ-test .................................... SUCCESS [ 14.574 s]
[INFO] hudi-integ-test-bundle ............................. SUCCESS [ 33.876 s]
[INFO] hudi-examples ...................................... SUCCESS [  9.117 s]
[INFO] hudi-flink_2.12 .................................... SUCCESS [  5.476 s]
[INFO] hudi-kafka-connect ................................. SUCCESS [  3.303 s]
[INFO] hudi-flink-bundle_2.12 ............................. SUCCESS [ 21.331 s]
[INFO] hudi-kafka-connect-bundle .......................... SUCCESS [ 15.754 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:34 min
[INFO] Finished at: 2021-12-02T12:06:21-08:00
[INFO] ------------------------------------------------------------------------ {code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-2919
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-2261

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by examining the Maven reactor and the Hudi modules listed in the build log; the issue names no files, tests, or implementation entry points. Use the reported 6.5-minute build as the baseline, and treat completion as migrating the project to Bazel while preserving the listed build scope and improving incremental-build behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.