apache / apache/paimon

[Bug] Batch DELETE is a silent no-op on partial-update tables with remove-record-on-sequence-group

Open
#8,858 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

master

### Compute Engine

Spark

### Minimal reproduce step

```sql
CREATE TABLE t (id INT, g INT, v BIGINT) TBLPROPERTIES (
'primary-key' = 'id',
'bucket' = '2',
'merge-engine' = 'partial-update',
'fields.g.sequence-group' = 'v',
'partial-update.remove-record-on-sequence-group' = 'g');

INSERT INTO t VALUES (1, 1, 10), (2, 1, 20);
DELETE FROM t WHERE id = 1;
SELECT * FROM t;
-- both rows are still there, the DELETE silently has no effect
```

### What doesn't meet your expectations?

`validatePKUpsertDeletable` allows batch DELETE when `partial-update.remove-record-on-sequence-group` is set, so the statement goes through the pk-upsert delete path (scan matching rows, mark them with `-D` and write back). The `-D` rows carry the real field values including the sequence-group field, so I'd expect `retractWithSequenceGroup` to set `currentDeleteRow` and remove the row — but nothing is deleted and no error is raised.

Not sure whether this is a bug or expected (e.g. the delete row is required to carry a *larger* sequence value than the current one?). If it is expected, maybe `validatePKUpsertDeletable` should reject this case or the behavior should be documented, since a silently no-op DELETE is quite surprising.

Noticed while working on #8837 (the fast path there is gated to DEDUPLICATE, so it's unrelated to that change — reproducible on vanilla master).

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the Spark SQL reproduction in the issue and inspect validatePKUpsertDeletable and retractWithSequenceGroup to trace how the sequence-group field is handled by batch DELETE. Compare the expected delete semantics with the observed -D rows, then make the behavior delete the matching rows or reject the unsupported case explicitly, with a regression test for the reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.