Setup HDFS in GitHub CI
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
To execute unit tests with Java 17, we can't use MiniDFSCluster and have to rely on external HDFS.
Utils to use external HDFS:
https://github.com/apache/hudi/blob/2a0223933884cb044e7aa56f205cae926358a030/hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestUtils.java#L241
Currently, we have docker image set up and those Java 17 tests would be executed there ([https://github.com/apache/hudi/blob/2a0223933884cb044e7aa56f205cae926358a030/.github/workflows/bot.yml#L209)]
A more ideal solution would be setting up HDFS in GitHub CI directly, for example: https://github.com/beyondstorage/setup-hdfs
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6600
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with .github/workflows/bot.yml around line 209 and compare its Docker-based Java 17 test setup with the external HDFS helpers in hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestUtils.java around line 241. Review the setup-hdfs example linked in the issue and identify how HDFS can run directly in GitHub CI. Done means the Java 17 unit tests use CI-provided HDFS without relying on the existing Docker test environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, java
- Domain
- ci-cd, infrastructure, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100