apache / apache/paimon

[Bug] Aggregation table retract issue

Open
#4,670 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

master

### Compute Engine

core

### Minimal reproduce step

Have writen some unit tests to demonstrate this issue, see https://github.com/apache/paimon/compare/master...zhongyujiang:incubator-paimon:gh/retract-issue

### What doesn't meet your expectations?

The retraction result of the agg table might be incorrect:
- the early aggregation before write will merge retract records with the same key into a single +I record
- during compaction, retract records are eliminated, even if there are still records with the same key at lower levels in the LSM tree that are supposed to be retracted

These two points will cause retraction on agg table to fail, please see the unit tests I wrote in the above link.

It seems we need to handle retracts in a similar way to how we handle delete tombstones in LSM, but that would result in both an insert and a delete record for the same key existing within a single sorted run, which might be challenging to handle I guess?

cc @JingsongLi @tsreaper could you please take a look? Thanks

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the unit tests in the referenced comparison and trace aggregation-table handling for retract records before write and during LSM compaction. Done means the tests demonstrate correct retraction results without eliminating records that still need to be retracted, and the relevant unit tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.