apache / apache/hudi

[SUPPORT] Hive Sync tool fails to sync Hoodi table written using Flink 1.16 to HMS

Open
#8,848 21 comments 0 reactions 0 assignees View on GitHub
component:catalog-sync issue:class-conflict
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**
Hive sync tool fails with the following stacktrace

```
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/calcite/plan/RelOptRule
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC(IMetaStoreClientUtil.java:40)
at org.apache.hudi.hive.HoodieHiveSyncClient.(HoodieHiveSyncClient.java:88)
at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:122)
at org.apache.hudi.hive.HiveSyncTool.(HiveSyncTool.java:116)
at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:482)
Caused by: java.lang.ClassNotFoundException: org.apache.calcite.plan.RelOptRule
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.C
```

**To Reproduce**
Write hudi tables using Flink 1.16, run Hive Sync tool on the local box to sync metadata to HMS specify `sync-mode` as `hms` and provide required parameters

**Expected behavior**

This should sync hoodie table to HMS

**Environment Description**

* Hudi version : 0.13.0

* Spark version :

* Hive version : 3.1.2

* Hadoop version : 2.10.2

* Storage (HDFS/S3/GCS..) : ABFS ( Azure )

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reported stack trace at org.apache.hudi.hive.util.IMetaStoreClientUtil.getMSC, then follow the HiveSyncTool and HoodieHiveSyncClient entry points. Reproduce the Hive sync command against HMS with a table written by Flink 1.16 and identify the missing Calcite dependency; done means the hoodie table syncs to HMS without the NoClassDefFoundError.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.