[Bug] Not sure if this is a bug but when you used fixed bucketing with dedicated compaction job is it expected for primary key to have partition column as well ?
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
**paimon 1.1.1**
**my partition lets say is done by age column and my primary key is an name column**
Is it expected to get this error:
Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: java.lang.IllegalArgumentException: You should use dynamic bucket (bucket = -1) mode in cross partition update case (Primary key constraint [name] not include all partition fields [age]).
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:373)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113)
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)
... 4 moren
### Compute Engine
flink 1.20
### Minimal reproduce step
write data with with name and age column keep the name as primary key and age as partition key
Schema.Builder schemaBuilder = Schema.newBuilder();
schemaBuilder.primaryKey("name");
schemaBuilder.partitionKeys("age");
schemaBuilder.column("name", DataTypes.STRING());
schemaBuilder.column("age", DataTypes.INT());
schemaBuilder.option("deletion-vectors.enabled", "true");
schemaBuilder.option("write-only", "true");
schemaBuilder.option("bucket-key", "name");
schemaBuilder.option("bucket", "4");
not able to run dedicated compaction:
tableEnv.executeSql(
"CALL sys.compact(" +
"'dedicated-employee.user'" +
")"
);
### What doesn't meet your expectations?
should be able to run compaction at a global level and not partition level
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Schema.Builder setup using name as the primary key and age as the partition key, then reproduce the failure through the CALL sys.compact('dedicated-employee.user') entry point. Check how dedicated compaction handles cross-partition updates and whether global compaction is supported. Done means the expected behavior is documented or the reported compaction path works without the shown constraint error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100