Hive-enabled SparkSession cannot bootstrap a metastore in hudi-spark tests (hive 2.3.10 vs libthrift 0.12.0)
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Describe the problem
Any test in `hudi-spark` that builds a session with `.enableHiveSupport()` (embedded Derby metastore) aborts on every JDK with:
```
java.lang.NoClassDefFoundError: org/apache/thrift/transport/layered/TFramedTransport
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getClass(MetaStoreUtils.java:1708)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:131)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3607)
...
```
Verified on current master with JDK 17 (`mvn test -Dspark3.5 -Pjava17 -DwildcardSuites=...` on `hudi-spark-datasource/hudi-spark`). The failure is JDK-independent: it is a classpath conflict, not a Java 9+ metastore incompatibility.
### Root cause
The root pom pins `hive.version=2.3.10`, whose metastore client references the thrift 0.13+ `layered` package, but the `hudi-spark` test classpath resolves `org.apache.thrift:libthrift:0.12.0` via Spark 3.5's dependencies (the pom's `thrift.version=0.13.0` property does not win on this path; confirmed with `mvn dependency:tree`).
### Impact
- Blocks any HMS-backed-catalog test coverage in the Spark modules. This is the gap behind HUDI-9678: no SQL-write test runs against a hive-backed session catalog, and PR #13748 could not add one because of this.
- `TestSparkCatalogMetaStoreClient` (added in #18672) aborts with the same error when executed (verified locally). The breakage is invisible today because `org.apache.spark.sql.hive` is missing from the scala-suite wildcard filters (`SCALA_TEST_DML_FILTER` / `SCALA_TEST_OTHERS_FILTER` in `.github/workflows/bot.yml`, and the job6 suite list in `azure-pipelines-20230430.yml`), so that suite never runs in CI.
### Fix direction
1. Manage libthrift to a hive-2.3.10-compatible version (>= 0.13) on the test classpath, or align the hive pin with Spark 3.5's hive 2.3.9.
2. Add `org.apache.spark.sql.hive` to the CI scala-suite filters so scalatest suites there actually execute (JUnit tests in that package are unaffected; they are selected via `-Dtest` filters).
3. With 1 and 2 in place, hive-enabled SQL test coverage (the intent of #13748) can be added.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the root pom and run the stated Maven test and dependency:tree command in hudi-spark-datasource/hudi-spark to confirm the Hive and libthrift conflict. Then inspect SCALA_TEST_DML_FILTER and SCALA_TEST_OTHERS_FILTER in .github/workflows/bot.yml and the job6 suite list in azure-pipelines-20230430.yml. Done means Hive-enabled Spark tests bootstrap successfully and the intended org.apache.spark.sql.hive suites execute in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- build-system, ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100