Migrate Hudi to proper Incremental Build System (Bazel)
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Even though Maven claims that it supports incremental builds, that is not what i personally observed in a day to day operations: even single file change trickles down into *whole project* being rebuilt which is currently taking about {*}6.5 mins{*}.
This is a substantial productivity drain, so migration to more modern Build System (like Bazel), while not easy, should bring the fruits quite fast, paying itself off well over by just how much time we would be saving on not waiting for these builds to complete.
{code:java}
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hudi 0.11.0-SNAPSHOT:
[INFO]
[INFO] Hudi ............................................... SUCCESS [ 5.150 s]
[INFO] hudi-common ........................................ SUCCESS [ 18.957 s]
[INFO] hudi-aws ........................................... SUCCESS [ 4.052 s]
[INFO] hudi-timeline-service .............................. SUCCESS [ 5.437 s]
[INFO] hudi-client ........................................ SUCCESS [ 0.291 s]
[INFO] hudi-client-common ................................. SUCCESS [ 13.902 s]
[INFO] hudi-hadoop-mr ..................................... SUCCESS [ 5.227 s]
[INFO] hudi-spark-client .................................. SUCCESS [ 23.578 s]
[INFO] hudi-sync-common ................................... SUCCESS [ 1.630 s]
[INFO] hudi-hive-sync ..................................... SUCCESS [ 10.545 s]
[INFO] hudi-spark-datasource .............................. SUCCESS [ 0.090 s]
[INFO] hudi-spark-common_2.12 ............................. SUCCESS [ 17.497 s]
[INFO] hudi-spark3_2.12 ................................... SUCCESS [ 10.558 s]
[INFO] hudi-spark_2.12 .................................... SUCCESS [ 46.709 s]
[INFO] hudi-utilities_2.12 ................................ SUCCESS [ 9.061 s]
[INFO] hudi-utilities-bundle_2.12 ......................... SUCCESS [ 18.046 s]
[INFO] hudi-cli ........................................... SUCCESS [ 16.890 s]
[INFO] hudi-java-client ................................... SUCCESS [ 3.887 s]
[INFO] hudi-flink-client .................................. SUCCESS [ 10.512 s]
[INFO] hudi-spark2_2.12 ................................... SUCCESS [ 15.522 s]
[INFO] hudi-dla-sync ...................................... SUCCESS [ 2.017 s]
[INFO] hudi-sync .......................................... SUCCESS [ 0.075 s]
[INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [ 6.188 s]
[INFO] hudi-hive-sync-bundle .............................. SUCCESS [ 1.783 s]
[INFO] hudi-spark3-bundle_2.12 ............................ SUCCESS [ 17.597 s]
[INFO] hudi-presto-bundle ................................. SUCCESS [ 4.982 s]
[INFO] hudi-timeline-server-bundle ........................ SUCCESS [ 6.391 s]
[INFO] hudi-hadoop-docker ................................. SUCCESS [ 2.110 s]
[INFO] hudi-hadoop-base-docker ............................ SUCCESS [ 1.404 s]
[INFO] hudi-hadoop-namenode-docker ........................ SUCCESS [ 1.124 s]
[INFO] hudi-hadoop-datanode-docker ........................ SUCCESS [ 1.090 s]
[INFO] hudi-hadoop-history-docker ......................... SUCCESS [ 1.098 s]
[INFO] hudi-hadoop-hive-docker ............................ SUCCESS [ 2.019 s]
[INFO] hudi-hadoop-sparkbase-docker ....................... SUCCESS [ 0.993 s]
[INFO] hudi-hadoop-sparkmaster-docker ..................... SUCCESS [ 1.106 s]
[INFO] hudi-hadoop-sparkworker-docker ..................... SUCCESS [ 1.063 s]
[INFO] hudi-hadoop-sparkadhoc-docker ...................... SUCCESS [ 1.031 s]
[INFO] hudi-hadoop-presto-docker .......................... SUCCESS [ 1.103 s]
[INFO] hudi-integ-test .................................... SUCCESS [ 14.574 s]
[INFO] hudi-integ-test-bundle ............................. SUCCESS [ 33.876 s]
[INFO] hudi-examples ...................................... SUCCESS [ 9.117 s]
[INFO] hudi-flink_2.12 .................................... SUCCESS [ 5.476 s]
[INFO] hudi-kafka-connect ................................. SUCCESS [ 3.303 s]
[INFO] hudi-flink-bundle_2.12 ............................. SUCCESS [ 21.331 s]
[INFO] hudi-kafka-connect-bundle .......................... SUCCESS [ 15.754 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:34 min
[INFO] Finished at: 2021-12-02T12:06:21-08:00
[INFO] ------------------------------------------------------------------------ {code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-2919
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-2261
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by examining the Maven reactor and the Hudi modules listed in the build log; the issue names no files, tests, or implementation entry points. Use the reported 6.5-minute build as the baseline, and treat completion as migrating the project to Bazel while preserving the listed build scope and improving incremental-build behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100