[Bug] Aggregation table retract issue
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
master
### Compute Engine
core
### Minimal reproduce step
Have writen some unit tests to demonstrate this issue, see https://github.com/apache/paimon/compare/master...zhongyujiang:incubator-paimon:gh/retract-issue
### What doesn't meet your expectations?
The retraction result of the agg table might be incorrect:
- the early aggregation before write will merge retract records with the same key into a single +I record
- during compaction, retract records are eliminated, even if there are still records with the same key at lower levels in the LSM tree that are supposed to be retracted
These two points will cause retraction on agg table to fail, please see the unit tests I wrote in the above link.
It seems we need to handle retracts in a similar way to how we handle delete tombstones in LSM, but that would result in both an insert and a delete record for the same key existing within a single sorted run, which might be challenging to handle I guess?
cc @JingsongLi @tsreaper could you please take a look? Thanks
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the unit tests in the referenced comparison and trace aggregation-table handling for retract records before write and during LSM compaction. Done means the tests demonstrate correct retraction results without eliminating records that still need to be retracted, and the relevant unit tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100