apache / apache/paimon

[Bug] Spark engine fails to generate +U events in changelog when using partial-update/aggregation and lookup changelog-producer

Open
#6,946 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

1.3.1

### Compute Engine

Spark 3.5.2

### Minimal reproduce step

CREATE TABLE test_partial_lookup (
id INT,
name STRING,
age INT
) TBLPROPERTIES (
'merge-engine' = 'partial-update',
'changelog-producer' = 'lookup',
'primary-key' = 'id'
);

spark.sql("insert into test_partial_lookup values(1, "test", 20)").show(10, False)

# only +I rowkind
spark.sql("select * from `test_partial_lookup$audit_log`").show(10, False)
# check compaction event
spark.sql("select * from `test_partial_lookup$snapshots` order by snapshot_id desc").show(10, False)

# update the available record to get the updated event
spark.sql("update test_partial_lookup set name = 'test1' where id = 1").show(10, False)

# check updated record
spark.sql("select * from test_partial_lookup").show(10, False)
# only +I rowkind after updating
spark.sql("select * from `test_partial_lookup$audit_log`").show(10, False)
# check compaction event
spark.sql("select * from `test_partial_lookup$snapshots` order by snapshot_id desc").show(10, False)

### What doesn't meet your expectations?

Yes, I want to capture the updated event from the change log using spark

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the SQL reproduction against Paimon 1.3.1 with Spark 3.5.2, then compare the audit_log and snapshots results before and after the update. Done means the partial-update/aggregation case with the lookup changelog producer records the updated event with the expected +U row kind.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.