apache / apache/paimon

[Bug] Not sure if this is a bug but when you used fixed bucketing with dedicated compaction job is it expected for primary key to have partition column as well ?

Open
#5,920 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

**paimon 1.1.1**

**my partition lets say is done by age column and my primary key is an name column**
Is it expected to get this error:

Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: java.lang.IllegalArgumentException: You should use dynamic bucket (bucket = -1) mode in cross partition update case (Primary key constraint [name] not include all partition fields [age]).
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:373)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113)
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)
... 4 moren

### Compute Engine

flink 1.20

### Minimal reproduce step

write data with with name and age column keep the name as primary key and age as partition key
Schema.Builder schemaBuilder = Schema.newBuilder();
schemaBuilder.primaryKey("name");
schemaBuilder.partitionKeys("age");
schemaBuilder.column("name", DataTypes.STRING());
schemaBuilder.column("age", DataTypes.INT());
schemaBuilder.option("deletion-vectors.enabled", "true");
schemaBuilder.option("write-only", "true");
schemaBuilder.option("bucket-key", "name");
schemaBuilder.option("bucket", "4");

not able to run dedicated compaction:
tableEnv.executeSql(
"CALL sys.compact(" +
"'dedicated-employee.user'" +
")"
);

### What doesn't meet your expectations?

should be able to run compaction at a global level and not partition level

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Schema.Builder setup using name as the primary key and age as the partition key, then reproduce the failure through the CALL sys.compact('dedicated-employee.user') entry point. Check how dedicated compaction handles cross-partition updates and whether global compaction is supported. Done means the expected behavior is documented or the reported compaction path works without the shown constraint error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.