apache / apache/hudi

Hive-enabled SparkSession cannot bootstrap a metastore in hudi-spark tests (hive 2.3.10 vs libthrift 0.12.0)

Open
#19,296 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Describe the problem

Any test in `hudi-spark` that builds a session with `.enableHiveSupport()` (embedded Derby metastore) aborts on every JDK with:

```
java.lang.NoClassDefFoundError: org/apache/thrift/transport/layered/TFramedTransport
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getClass(MetaStoreUtils.java:1708)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:131)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3607)
...
```

Verified on current master with JDK 17 (`mvn test -Dspark3.5 -Pjava17 -DwildcardSuites=...` on `hudi-spark-datasource/hudi-spark`). The failure is JDK-independent: it is a classpath conflict, not a Java 9+ metastore incompatibility.

### Root cause

The root pom pins `hive.version=2.3.10`, whose metastore client references the thrift 0.13+ `layered` package, but the `hudi-spark` test classpath resolves `org.apache.thrift:libthrift:0.12.0` via Spark 3.5's dependencies (the pom's `thrift.version=0.13.0` property does not win on this path; confirmed with `mvn dependency:tree`).

### Impact

- Blocks any HMS-backed-catalog test coverage in the Spark modules. This is the gap behind HUDI-9678: no SQL-write test runs against a hive-backed session catalog, and PR #13748 could not add one because of this.
- `TestSparkCatalogMetaStoreClient` (added in #18672) aborts with the same error when executed (verified locally). The breakage is invisible today because `org.apache.spark.sql.hive` is missing from the scala-suite wildcard filters (`SCALA_TEST_DML_FILTER` / `SCALA_TEST_OTHERS_FILTER` in `.github/workflows/bot.yml`, and the job6 suite list in `azure-pipelines-20230430.yml`), so that suite never runs in CI.

### Fix direction

1. Manage libthrift to a hive-2.3.10-compatible version (>= 0.13) on the test classpath, or align the hive pin with Spark 3.5's hive 2.3.9.
2. Add `org.apache.spark.sql.hive` to the CI scala-suite filters so scalatest suites there actually execute (JUnit tests in that package are unaffected; they are selected via `-Dtest` filters).
3. With 1 and 2 in place, hive-enabled SQL test coverage (the intent of #13748) can be added.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the root pom and run the stated Maven test and dependency:tree command in hudi-spark-datasource/hudi-spark to confirm the Hive and libthrift conflict. Then inspect SCALA_TEST_DML_FILTER and SCALA_TEST_OTHERS_FILTER in .github/workflows/bot.yml and the job6 suite list in azure-pipelines-20230430.yml. Done means Hive-enabled Spark tests bootstrap successfully and the intended org.apache.spark.sql.hive suites execute in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
build-system, ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.