apache / apache/paimon

[Bug] An exception occurs when flink task writes to paimon secondary partition

Open
#4,949 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon-0.9.0
paimon-1.0.0

### Compute Engine

Flink-1.18.1

### Minimal reproduce step

Pamon table creation script:
`
`CREATE TABLE `paimon_test.user_behavior_fusion_di_test_13`(
`seq` int COMMENT 'from deserializer',
`meta_id` int COMMENT 'from deserializer',
`service_id` string COMMENT 'from deserializer',
`client_timestamp` bigint COMMENT 'from deserializer',
`server_timestamp` bigint COMMENT 'from deserializer',
`props` map COMMENT 'from deserializer',
`channel_name` string COMMENT 'from deserializer',
`os_name` string COMMENT 'from deserializer',
`version_no` string COMMENT 'from deserializer',
`session_id` string COMMENT 'from deserializer',
`device_id` string COMMENT 'from deserializer',
`device_name` string COMMENT 'from deserializer',
`device_type_name` string COMMENT 'from deserializer',
`user_id` bigint COMMENT 'from deserializer',
`app_id` int COMMENT 'from deserializer',
`bid` int COMMENT 'from deserializer',
`extra_attribute` map COMMENT 'from deserializer',
`client_send_timestamp` bigint COMMENT 'from deserializer',
`insert_timestamp` timestamp COMMENT 'from deserializer',
`uncommon_map` map COMMENT 'from deserializer',
`random_column` int COMMENT 'from deserializer',
`data_version` int COMMENT 'from deserializer',
`location_code` bigint COMMENT 'from deserializer',
`duration_time` bigint COMMENT 'from deserializer',
`device_id_code` bigint COMMENT 'from deserializer',
`is_lock_exposed` boolean COMMENT 'from deserializer',
`from_back` int COMMENT 'from deserializer')
PARTITIONED BY (
`dt` string COMMENT '日期, yyyyMMdd',
`event_id` string COMMENT '事件类型')
ROW FORMAT SERDE
'org.apache.paimon.hive.PaimonSerDe'
STORED BY
'org.apache.paimon.hive.PaimonStorageHandler'
WITH SERDEPROPERTIES (
'serialization.format'='1')
TBLPROPERTIES (
'bucket'='-1',
'bucketing_version'='2',
'file-index.bitmap.columns'='app_id,meta_id,bid',
'file.format'='parquet',
'num-sorted-run.stop-trigger'='30',
'parquet.compression'='zstd',
'partition.expiration-check-interval'='1d',
'partition.expiration-time'='365d',
'partition.timestamp-formatter'='yyyyMMdd',
'partition.timestamp-pattern'='$dt',
'snapshot.expire.limit'='8',
'snapshot.num-retained.min'='16',
'snapshot.time-retained'='6h',
'sort-spill-threshold'='10',
'target-file-size'='1GB',
'transient_lastDdlTime'='1737100343',
'write-buffer-size'='512MB',
'write-manifest-cache'='1GB')`。
`

The task will generally succeed in the first checkpoint. In terms of performance, it will be written successfully when there is no partition information, and an exception will be reported later.

### What doesn't meet your expectations?

`2025-01-20 11:12:02
java.lang.RuntimeException: MetaException(message:Exception thrown when executing query : SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS `NUCLEUS_TYPE`,`A0`.`CREATE_TIME`,`A0`.`LAST_ACCESS_TIME`,`A0`.`PART_NAME`,`A0`.`PART_ID` FROM `PARTITIONS` `A0` LEFT OUTER JOIN `TBLS` `B0` ON `A0`.`TBL_ID` = `B0`.`TBL_ID` LEFT OUTER JOIN `DBS` `C0` ON `B0`.`DB_ID` = `C0`.`DB_ID` WHERE `B0`.`TBL_NAME` = ? AND `C0`.`NAME` = ? AND `A0`.`PART_NAME` = ? AND `C0`.`CTLG_NAME` = ?)
at org.apache.paimon.metastore.AddPartitionCommitCallback.addPartitions(AddPartitionCommitCallback.java:95)
at org.apache.paimon.metastore.AddPartitionCommitCallback.retry(AddPartitionCommitCallback.java:76)
at org.apache.paimon.operation.FileStoreCommitImpl.lambda$filterCommitted$0(FileStoreCommitImpl.java:247)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.apache.paimon.operation.FileStoreCommitImpl.filterCommitted(FileStoreCommitImpl.java:247)
at org.apache.paimon.table.sink.TableCommitImpl.filterAndCommitMultiple(TableCommitImpl.java:244)
at org.apache.paimon.flink.sink.StoreCommitter.filterAndCommit(StoreCommitter.java:119)
at org.apache.paimon.flink.sink.Committer.filterAndCommit(Committer.java:60)
at org.apache.paimon.flink.sink.RestoreAndFailCommittableStateManager.recover(RestoreAndFailCommittableStateManager.java:82)
at org.apache.paimon.flink.sink.RestoreAndFailCommittableStateManager.initializeState(RestoreAndFailCommittableStateManager.java:77)
at org.apache.paimon.flink.sink.CommitterOperator.initializeState(CommitterOperator.java:142)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:753)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:728)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:693)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:922)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.lang.Thread.run(Thread.java:750)
Caused by: MetaException(message:Exception thrown when executing query : SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS `NUCLEUS_TYPE`,`A0`.`CREATE_TIME`,`A0`.`LAST_ACCESS_TIME`,`A0`.`PART_NAME`,`A0`.`PART_ID` FROM `PARTITIONS` `A0` LEFT OUTER JOIN `TBLS` `B0` ON `A0`.`TBL_ID` = `B0`.`TBL_ID` LEFT OUTER JOIN `DBS` `C0` ON `B0`.`DB_ID` = `C0`.`DB_ID` WHERE `B0`.`TBL_NAME` = ? AND `C0`.`NAME` = ? AND `A0`.`PART_NAME` = ? AND `C0`.`CTLG_NAME` = ?)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_partitions_req(ThriftHiveMetastore.java:2488)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_partitions_req(ThriftHiveMetastore.java:2475)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.add_partitions(HiveMetaStoreClient.java:695)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
at com.sun.proxy.$Proxy27.add_partitions(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773)
at com.sun.proxy.$Proxy28.add_partitions(Unknown Source)
at org.apache.paimon.hive.HiveMetastoreClient.lambda$addPartitions$3(HiveMetastoreClient.java:107)
at org.apache.paimon.client.ClientPool$ClientPoolImpl.lambda$execute$0(ClientPool.java:80)
at org.apache.paimon.client.ClientPool$ClientPoolImpl.run(ClientPool.java:68)
at org.apache.paimon.client.ClientPool$ClientPoolImpl.execute(ClientPool.java:77)
at org.apache.paimon.hive.pool.CachedClientPool.execute(CachedClientPool.java:139)
at org.apache.paimon.hive.HiveMetastoreClient.addPartitions(HiveMetastoreClient.java:107)
at org.apache.paimon.metastore.AddPartitionCommitCallback.addPartitions(AddPartitionCommitCallback.java:88)
... 22 more
`

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with AddPartitionCommitCallback.java and HiveMetastoreClient.java, then trace the partition-addition path shown in the stack trace. Reproduce the failure with Flink 1.18.1 and the provided partitioned-table setup; done means writing to a secondary partition and checkpoint recovery complete without the Hive metastore exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.