apache / apache/paimon

[Bug] When the version 0.7 is updated, hive cannot be connected

Open
#3,153 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon 0.7.0

### Compute Engine

Flink 1.17.2

### Minimal reproduce step

/bin/flink run \
-Dpipeline.name=test_paimon \
-Dexecution.checkpointing.interval=10000 \
lib/paimon-flink-action-0.7.0-incubating.jar \
mysql_sync_database\
--warehouse hdfs://cdh1:8020/user/hive/warehouse/paimon \
--database test3 \
--mode combined\
--mysql-conf hostname=cdh1 \
--mysql-conf username=root \
--mysql-conf password=123456 \
--mysql-conf database-name=test \
--catalog-conf metastore=hive \
--catalog-conf uri=thrift://cdh1:9083 \
--table-conf bucket=4 \
--table-conf changelog-producer=input \
--table-conf sink.parallelism=4 \
--including_tables '.*'

### What doesn't meet your expectations?

The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: java.lang.reflect.InvocationTargetException
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095)
at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157)
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.paimon.hive.RetryingMetaStoreClientFactory.createClient(RetryingMetaStoreClientFactory.java:146)
at org.apache.paimon.hive.HiveCatalog.createClient(HiveCatalog.java:600)
at org.apache.paimon.hive.HiveCatalog.(HiveCatalog.java:147)
at org.apache.paimon.hive.HiveCatalog.createHiveCatalog(HiveCatalog.java:692)
at org.apache.paimon.hive.HiveCatalogFactory.create(HiveCatalogFactory.java:53)
at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:76)
at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66)
at org.apache.paimon.flink.FlinkCatalogFactory.createPaimonCatalog(FlinkCatalogFactory.java:80)
at org.apache.paimon.flink.action.ActionBase.(ActionBase.java:59)
at org.apache.paimon.flink.action.cdc.SynchronizationActionBase.(SynchronizationActionBase.java:71)
at org.apache.paimon.flink.action.cdc.SyncDatabaseActionBase.(SyncDatabaseActionBase.java:61)
at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseAction.(MySqlSyncDatabaseAction.java:107)
at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.create(MySqlSyncDatabaseActionFactory.java:61)
at org.apache.paimon.flink.action.ActionFactory.createAction(ActionFactory.java:82)
at org.apache.paimon.flink.action.FlinkActions.main(FlinkActions.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
... 12 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.paimon.hive.RetryingMetaStoreClientFactory.lambda$static$7(RetryingMetaStoreClientFactory.java:111)
at org.apache.paimon.hive.RetryingMetaStoreClientFactory.createClient(RetryingMetaStoreClientFactory.java:144)
... 31 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1775)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:86)
... 37 more
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean)
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771)
... 40 more

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Run the provided Flink 1.17.2 command against Paimon 0.7.0 and reproduce the Hive metastore constructor error. Start with org.apache.paimon.hive.RetryingMetaStoreClientFactory and HiveCatalog, then inspect the Hive and Hadoop dependency versions involved. Done means the mysql_sync_database action connects to the Hive catalog without the NoSuchMethodException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.