[Bug] CALL sys.expire_partitions failed when using hive metastore and setting 'metastore.partitioned-table' = 'false'.
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
release-1.0
### Compute Engine
spark-3.2.0
### Minimal reproduce step
step 1: To create a date partitioned table, set 'metastore.partitioned-table' = 'false'
step 2: Write the test data of the last N days
step 3: CALL sys.expire_partitions(table => 'db.tb', expiration_time => '1 d', timestamp_formatter => 'yyyy-MM-dd');
### What doesn't meet your expectations?
An exception is thrown when sys.expire_partitions is executed and no snapshot is generated, so the partitions that need to expire after the snapshot expires will not be physically deleted.
### Anything else?
Exception information:
spark-sql> CALL sys.expire_partitions(table => 'db.tb', expiration_time => '1 d', timestamp_formatter => 'yyyy-MM-dd');
25/01/08 17:50:18 ERROR SparkSQLDriver: Failed in [CALL sys.expire_partitions(table => 'db.tb', expiration_time => '1 d', timestamp_formatter => 'yyyy-MM-dd')]
java.lang.RuntimeException: MetaException(message:Invalid partition key & values; keys [], values [2025-01-01, 15, ])
at org.apache.paimon.operation.PartitionExpire.deleteMetastorePartitions(PartitionExpire.java:175)
at org.apache.paimon.operation.PartitionExpire.doExpire(PartitionExpire.java:162)
at org.apache.paimon.operation.PartitionExpire.expire(PartitionExpire.java:139)
at org.apache.paimon.operation.PartitionExpire.expire(PartitionExpire.java:109)
at org.apache.paimon.spark.procedure.ExpirePartitionsProcedure.lambda$call$2(ExpirePartitionsProcedure.java:115)
at org.apache.paimon.spark.procedure.BaseProcedure.execute(BaseProcedure.java:88)
at org.apache.paimon.spark.procedure.BaseProcedure.modifyPaimonTable(BaseProcedure.java:78)
at org.apache.paimon.spark.procedure.ExpirePartitionsProcedure.call(ExpirePartitionsProcedure.java:87)
at org.apache.paimon.spark.execution.PaimonCallExec.run(PaimonCallExec.scala:32)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with PartitionExpire.java, especially deleteMetastorePartitions, and follow the call from ExpirePartitionsProcedure.java. Reproduce the issue with Spark 3.2.0, a date-partitioned table using metastore.partitioned-table=false, and the shown CALL statement. Done means sys.expire_partitions completes without the invalid partition-key exception and physically deletes expired partitions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100