[Bug] When the version 0.7 is updated, hive cannot be connected
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
paimon 0.7.0
### Compute Engine
Flink 1.17.2
### Minimal reproduce step
/bin/flink run \
-Dpipeline.name=test_paimon \
-Dexecution.checkpointing.interval=10000 \
lib/paimon-flink-action-0.7.0-incubating.jar \
mysql_sync_database\
--warehouse hdfs://cdh1:8020/user/hive/warehouse/paimon \
--database test3 \
--mode combined\
--mysql-conf hostname=cdh1 \
--mysql-conf username=root \
--mysql-conf password=123456 \
--mysql-conf database-name=test \
--catalog-conf metastore=hive \
--catalog-conf uri=thrift://cdh1:9083 \
--table-conf bucket=4 \
--table-conf changelog-producer=input \
--table-conf sink.parallelism=4 \
--including_tables '.*'
### What doesn't meet your expectations?
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: java.lang.reflect.InvocationTargetException
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095)
at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157)
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.paimon.hive.RetryingMetaStoreClientFactory.createClient(RetryingMetaStoreClientFactory.java:146)
at org.apache.paimon.hive.HiveCatalog.createClient(HiveCatalog.java:600)
at org.apache.paimon.hive.HiveCatalog.(HiveCatalog.java:147)
at org.apache.paimon.hive.HiveCatalog.createHiveCatalog(HiveCatalog.java:692)
at org.apache.paimon.hive.HiveCatalogFactory.create(HiveCatalogFactory.java:53)
at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:76)
at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66)
at org.apache.paimon.flink.FlinkCatalogFactory.createPaimonCatalog(FlinkCatalogFactory.java:80)
at org.apache.paimon.flink.action.ActionBase.(ActionBase.java:59)
at org.apache.paimon.flink.action.cdc.SynchronizationActionBase.(SynchronizationActionBase.java:71)
at org.apache.paimon.flink.action.cdc.SyncDatabaseActionBase.(SyncDatabaseActionBase.java:61)
at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseAction.(MySqlSyncDatabaseAction.java:107)
at org.apache.paimon.flink.action.cdc.mysql.MySqlSyncDatabaseActionFactory.create(MySqlSyncDatabaseActionFactory.java:61)
at org.apache.paimon.flink.action.ActionFactory.createAction(ActionFactory.java:82)
at org.apache.paimon.flink.action.FlinkActions.main(FlinkActions.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
... 12 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.paimon.hive.RetryingMetaStoreClientFactory.lambda$static$7(RetryingMetaStoreClientFactory.java:111)
at org.apache.paimon.hive.RetryingMetaStoreClientFactory.createClient(RetryingMetaStoreClientFactory.java:144)
... 31 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1775)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:80)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:86)
... 37 more
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(org.apache.hadoop.hive.conf.HiveConf, java.lang.Boolean)
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1771)
... 40 more
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Run the provided Flink 1.17.2 command against Paimon 0.7.0 and reproduce the Hive metastore constructor error. Start with org.apache.paimon.hive.RetryingMetaStoreClientFactory and HiveCatalog, then inspect the Hive and Hadoop dependency versions involved. Done means the mysql_sync_database action connects to the Hive catalog without the NoSuchMethodException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100