Adapt Integration tests CI for Spark4
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Integration tests on Spark 4 do not work on existing hadoop/hive/spark docker images from apachehudi dockerhub repo:
{code:java}
java.lang.NoSuchFieldError: NVDIMM
at org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter.initClass(TestDFSHoodieTestSuiteWriterAdapter.java:67)
[ERROR] Errors:
[ERROR] TestDFSHoodieTestSuiteWriterAdapter.initClass:67->UtilitiesTestBase.initTestServices:152 » NoSuchField
[ERROR] TestFileDeltaInputWriter.initClass:62->UtilitiesTestBase.initTestServices:152 » NoSuchField
[ERROR] TestDFSAvroDeltaInputReader.initClass:47->UtilitiesTestBase.initTestServices:152 » NoSuchField
[ERROR] TestDFSHoodieDatasetInputReader.initClass:56->UtilitiesTestBase.initTestServices:152 » NoSuchField
[ERROR] Tests run: 21, Failures: 0, Errors: 4, Skipped: 2{code}
Need to investigate and fix (docker images or integration tests or both) and uncomment Spark 4 params in matrix of integration-tests-java17 CI pipeline.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9409
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Spark 4 failures in TestDFSHoodieTestSuiteWriterAdapter and the listed integration tests, focusing on the NVDIMM error from the existing Hadoop, Hive, and Spark Docker images. Inspect the integration-tests-java17 CI matrix and its Spark 4 parameters; done means the affected tests pass and Spark 4 is enabled in that matrix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, hadoop, java, spark
- Domain
- ci-cd, data-engineering, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100