apache / apache/paimon

[Bug] Contradictory partition-key equals are treated as a partition drop

Open Beginner friendly
#9,631 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

master, `475be566f` (2.1-SNAPSHOT).

### Compute Engine

Flink. `SupportsRowLevelOperationFlinkTableSink` is the only production user of this visitor.

### Minimal reproduce step

This one is a contract problem rather than something I could trigger from SQL, and I want to be upfront about that.

`OnlyPartitionKeyEqualVisitor` decides whether a DELETE can be executed by dropping partitions. It collects the equality literals into a map, one entry per partition key:

```java
public Boolean visitEqual(FieldRef fieldRef, Object literal) {
boolean contains = partitionKeys.contains(fieldRef.name());
if (contains) {
partitions.put(fieldRef.name(), literal.toString());
return true;
}
return false;
}
```

Two equality predicates on the same key therefore overwrite each other, and the conjunction still reports "droppable". Feeding it `pt = 'a' AND pt = 'b'`:

```java
OnlyPartitionKeyEqualVisitor visitor = new OnlyPartitionKeyEqualVisitor(Arrays.asList("pt", "dt"));
Predicate contradiction = PredicateBuilder.and(equal(pt, "a"), equal(pt, "b"));
contradiction.visit(visitor); // true
visitor.partitions(); // {pt=b}
```

The Flink sink reads that as a partition drop of `pt = 'b'` and deletes every row in that partition, while the predicate itself matches nothing.

What I could not do is get such a predicate to the sink from SQL. `DELETE FROM t WHERE pt = 'a' AND pt = 'b'` is folded to FALSE by Calcite's simplification before `applyDeleteFilters` runs, and `PredicateBuilder.and` does not merge equality predicates itself, so the visitor only sees two `Equal` leaves if something hands them over unfolded. So today the guarantee that this cannot happen lives in the planner, not in Paimon.

### What doesn't meet your expectations?

The visitor reduces a conjunction of constraints to a last-write-wins map, which loses the information that the constraints conflict. For a decision whose consequence is dropping a whole partition, that seems worth deciding inside Paimon rather than relying on an external optimizer to never pass a contradiction through. A future engine version, another engine binding, or a caller that builds the predicate directly would not have that protection.

### Anything else?

The same class already returns false for everything it does not understand (`visitIsNull`, `visitIn`, ranges, OR), so a conservative answer for a contradiction fits how it is written.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with OnlyPartitionKeyEqualVisitor, especially visitEqual and partitions(), then inspect its use by SupportsRowLevelOperationFlinkTableSink. Trace how a conjunction is classified and add coverage for contradictory equalities such as pt = 'a' AND pt = 'b'. Done means the contradiction is not considered droppable and cannot select a partition for deletion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.