[Bug] Spark engine fails to generate +U events in changelog when using partial-update/aggregation and lookup changelog-producer
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
1.3.1
### Compute Engine
Spark 3.5.2
### Minimal reproduce step
CREATE TABLE test_partial_lookup (
id INT,
name STRING,
age INT
) TBLPROPERTIES (
'merge-engine' = 'partial-update',
'changelog-producer' = 'lookup',
'primary-key' = 'id'
);
spark.sql("insert into test_partial_lookup values(1, "test", 20)").show(10, False)
# only +I rowkind
spark.sql("select * from `test_partial_lookup$audit_log`").show(10, False)
# check compaction event
spark.sql("select * from `test_partial_lookup$snapshots` order by snapshot_id desc").show(10, False)
# update the available record to get the updated event
spark.sql("update test_partial_lookup set name = 'test1' where id = 1").show(10, False)
# check updated record
spark.sql("select * from test_partial_lookup").show(10, False)
# only +I rowkind after updating
spark.sql("select * from `test_partial_lookup$audit_log`").show(10, False)
# check compaction event
spark.sql("select * from `test_partial_lookup$snapshots` order by snapshot_id desc").show(10, False)
### What doesn't meet your expectations?
Yes, I want to capture the updated event from the change log using spark
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the SQL reproduction against Paimon 1.3.1 with Spark 3.5.2, then compare the audit_log and snapshots results before and after the update. Done means the partial-update/aggregation case with the lookup changelog producer records the updated event with the expected +U row kind.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100