[Bug] full-compaction.delta-commits
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
77f130bde03381e3b26b86e4f0e95cff5c1dc622
### Compute Engine
Flink
### Minimal reproduce step
I created a pkey Paimon table with the following table properties with Iceberg compatibility enabled.
```
changelog-producer: lookup
full-compaction.delta-commits: 1
metadata.iceberg.storage-location: table-location
metadata.iceberg.manifest-legacy-version: true
metadata.iceberg.manifest-compression: snappy
metadata.iceberg.previous-versions-max: 24
metadata.iceberg.storage: hive-catalog
metadata.iceberg.hive-client-class: com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient
metadata.iceberg.glue.skip-archive: true
metadata.iceberg.uri: ""
metadata.iceberg.hive-conf-dir: ""
metadata.iceberg.hadoop-conf-dir: ""
```
I performed the inserts and update operations on the above created table.
### What doesn't meet your expectations?
After performing the updates, I don't see the updates getting reflected from Iceberg compatibility layer. I was following the source code, and it appears that the delta-commits trigger only happens when the changelog-producer is set to full-compaction. The delta-commits is handled only inside the [GlobalFullCompactionSinkWrite](https://github.com/apache/paimon/blob/77f130bde03381e3b26b86e4f0e95cff5c1dc622/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/GlobalFullCompactionSinkWrite.java#L172-L176), which only gets invoked when the [changelog-producer is full-compaction](https://github.com/apache/paimon/blob/77f130bde03381e3b26b86e4f0e95cff5c1dc622/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/StoreSinkWrite.java#L146-L146). Inside the docs as well, the full-compaction.delta-commits is only mentioned for [Full Compaction changelog-producer](https://paimon.apache.org/docs/1.2/primary-key-table/changelog-producer/#full-compaction).
When we have [changelog-producer as lookup](https://github.com/apache/paimon/blob/77f130bde03381e3b26b86e4f0e95cff5c1dc622/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/StoreSinkWrite.java#L160-L175), it relies on the [LookupSinkWrite](https://github.com/apache/paimon/blob/77f130bde03381e3b26b86e4f0e95cff5c1dc622/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/LookupSinkWrite.java#L36-L36) that doesn't take care of the delta-commit property.
### Anything else?
Based on the above behavior, I think of the following 2 solutions.
1. Either allow the `full-compaction.delta-commits` for other changelog producers as well. This will have a drawback increasing the cost of commit operations as full-compaction will be triggered everytime.
2. Improve the documentation, and not allow setting `full-compaction.delta-commits` property when changelog producer is different from `full-compaction`.
Let me know about your thoughts.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with GlobalFullCompactionSinkWrite.java and StoreSinkWrite.java at the linked lines, then compare LookupSinkWrite.java to the handling of full-compaction.delta-commits. Read the full-compaction changelog-producer documentation and the lookup behavior. Done means the issue's proposed behavior is resolved consistently in code and documentation, with regression coverage if the behavior changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100