[Bug] An exception occurs when flink task writes to paimon secondary partition
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
paimon-0.9.0
paimon-1.0.0
### Compute Engine
Flink-1.18.1
### Minimal reproduce step
Pamon table creation script:
`
`CREATE TABLE `paimon_test.user_behavior_fusion_di_test_13`(
`seq` int COMMENT 'from deserializer',
`meta_id` int COMMENT 'from deserializer',
`service_id` string COMMENT 'from deserializer',
`client_timestamp` bigint COMMENT 'from deserializer',
`server_timestamp` bigint COMMENT 'from deserializer',
`props` map COMMENT 'from deserializer',
`channel_name` string COMMENT 'from deserializer',
`os_name` string COMMENT 'from deserializer',
`version_no` string COMMENT 'from deserializer',
`session_id` string COMMENT 'from deserializer',
`device_id` string COMMENT 'from deserializer',
`device_name` string COMMENT 'from deserializer',
`device_type_name` string COMMENT 'from deserializer',
`user_id` bigint COMMENT 'from deserializer',
`app_id` int COMMENT 'from deserializer',
`bid` int COMMENT 'from deserializer',
`extra_attribute` map COMMENT 'from deserializer',
`client_send_timestamp` bigint COMMENT 'from deserializer',
`insert_timestamp` timestamp COMMENT 'from deserializer',
`uncommon_map` map COMMENT 'from deserializer',
`random_column` int COMMENT 'from deserializer',
`data_version` int COMMENT 'from deserializer',
`location_code` bigint COMMENT 'from deserializer',
`duration_time` bigint COMMENT 'from deserializer',
`device_id_code` bigint COMMENT 'from deserializer',
`is_lock_exposed` boolean COMMENT 'from deserializer',
`from_back` int COMMENT 'from deserializer')
PARTITIONED BY (
`dt` string COMMENT '日期, yyyyMMdd',
`event_id` string COMMENT '事件类型')
ROW FORMAT SERDE
'org.apache.paimon.hive.PaimonSerDe'
STORED BY
'org.apache.paimon.hive.PaimonStorageHandler'
WITH SERDEPROPERTIES (
'serialization.format'='1')
TBLPROPERTIES (
'bucket'='-1',
'bucketing_version'='2',
'file-index.bitmap.columns'='app_id,meta_id,bid',
'file.format'='parquet',
'num-sorted-run.stop-trigger'='30',
'parquet.compression'='zstd',
'partition.expiration-check-interval'='1d',
'partition.expiration-time'='365d',
'partition.timestamp-formatter'='yyyyMMdd',
'partition.timestamp-pattern'='$dt',
'snapshot.expire.limit'='8',
'snapshot.num-retained.min'='16',
'snapshot.time-retained'='6h',
'sort-spill-threshold'='10',
'target-file-size'='1GB',
'transient_lastDdlTime'='1737100343',
'write-buffer-size'='512MB',
'write-manifest-cache'='1GB')`。
`
The task will generally succeed in the first checkpoint. In terms of performance, it will be written successfully when there is no partition information, and an exception will be reported later.
### What doesn't meet your expectations?
`2025-01-20 11:12:02
java.lang.RuntimeException: MetaException(message:Exception thrown when executing query : SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS `NUCLEUS_TYPE`,`A0`.`CREATE_TIME`,`A0`.`LAST_ACCESS_TIME`,`A0`.`PART_NAME`,`A0`.`PART_ID` FROM `PARTITIONS` `A0` LEFT OUTER JOIN `TBLS` `B0` ON `A0`.`TBL_ID` = `B0`.`TBL_ID` LEFT OUTER JOIN `DBS` `C0` ON `B0`.`DB_ID` = `C0`.`DB_ID` WHERE `B0`.`TBL_NAME` = ? AND `C0`.`NAME` = ? AND `A0`.`PART_NAME` = ? AND `C0`.`CTLG_NAME` = ?)
at org.apache.paimon.metastore.AddPartitionCommitCallback.addPartitions(AddPartitionCommitCallback.java:95)
at org.apache.paimon.metastore.AddPartitionCommitCallback.retry(AddPartitionCommitCallback.java:76)
at org.apache.paimon.operation.FileStoreCommitImpl.lambda$filterCommitted$0(FileStoreCommitImpl.java:247)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.apache.paimon.operation.FileStoreCommitImpl.filterCommitted(FileStoreCommitImpl.java:247)
at org.apache.paimon.table.sink.TableCommitImpl.filterAndCommitMultiple(TableCommitImpl.java:244)
at org.apache.paimon.flink.sink.StoreCommitter.filterAndCommit(StoreCommitter.java:119)
at org.apache.paimon.flink.sink.Committer.filterAndCommit(Committer.java:60)
at org.apache.paimon.flink.sink.RestoreAndFailCommittableStateManager.recover(RestoreAndFailCommittableStateManager.java:82)
at org.apache.paimon.flink.sink.RestoreAndFailCommittableStateManager.initializeState(RestoreAndFailCommittableStateManager.java:77)
at org.apache.paimon.flink.sink.CommitterOperator.initializeState(CommitterOperator.java:142)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:753)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:728)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:693)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:922)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.lang.Thread.run(Thread.java:750)
Caused by: MetaException(message:Exception thrown when executing query : SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS `NUCLEUS_TYPE`,`A0`.`CREATE_TIME`,`A0`.`LAST_ACCESS_TIME`,`A0`.`PART_NAME`,`A0`.`PART_ID` FROM `PARTITIONS` `A0` LEFT OUTER JOIN `TBLS` `B0` ON `A0`.`TBL_ID` = `B0`.`TBL_ID` LEFT OUTER JOIN `DBS` `C0` ON `B0`.`DB_ID` = `C0`.`DB_ID` WHERE `B0`.`TBL_NAME` = ? AND `C0`.`NAME` = ? AND `A0`.`PART_NAME` = ? AND `C0`.`CTLG_NAME` = ?)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_partitions_req(ThriftHiveMetastore.java:2488)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_partitions_req(ThriftHiveMetastore.java:2475)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.add_partitions(HiveMetaStoreClient.java:695)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
at com.sun.proxy.$Proxy27.add_partitions(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773)
at com.sun.proxy.$Proxy28.add_partitions(Unknown Source)
at org.apache.paimon.hive.HiveMetastoreClient.lambda$addPartitions$3(HiveMetastoreClient.java:107)
at org.apache.paimon.client.ClientPool$ClientPoolImpl.lambda$execute$0(ClientPool.java:80)
at org.apache.paimon.client.ClientPool$ClientPoolImpl.run(ClientPool.java:68)
at org.apache.paimon.client.ClientPool$ClientPoolImpl.execute(ClientPool.java:77)
at org.apache.paimon.hive.pool.CachedClientPool.execute(CachedClientPool.java:139)
at org.apache.paimon.hive.HiveMetastoreClient.addPartitions(HiveMetastoreClient.java:107)
at org.apache.paimon.metastore.AddPartitionCommitCallback.addPartitions(AddPartitionCommitCallback.java:88)
... 22 more
`
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with AddPartitionCommitCallback.java and HiveMetastoreClient.java, then trace the partition-addition path shown in the stack trace. Reproduce the failure with Flink 1.18.1 and the provided partitioned-table setup; done means writing to a secondary partition and checkpoint recovery complete without the Hive metastore exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100