apache / apache/paimon-cpp

[Feature] Support changelog production

Đang mở
#174 0 bình luận 0 reaction 1 người được giao Được giao cho @lszskye Xem trên GitHub
enhancement
Ngôn ngữ chính
C++
Star
65
Fork
25
Merge trung bình
2 ngày 12 giờ
Pull request đã merge (30 ngày)
80

Mô tả

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.

### Motivation

Java Paimon supports producing changelog files for primary-key tables through the
`changelog-producer` option, including `input`, `lookup`, and `full-compaction`
modes.

Paimon C++ already contains part of the required infrastructure, such as:

- Parsing the `changelog-producer` option.
- Changelog file metadata in `DataIncrement` and `CompactIncrement`.
- Changelog manifest and snapshot commit support.
- Lookup changelog merge functions and changelog compaction rewriters.
- Streaming changelog scans and the `audit_log` and `binlog` system tables.

However, schema validation currently rejects every value except `none`, and the
writer does not produce changelog files. As a result, primary-key tables written
by Paimon C++ cannot provide the same changelog behavior as tables written by
Java Paimon.

Supporting Java-compatible changelog production would enable incremental
downstream consumption and improve interoperability between Java Paimon and
Paimon C++.

### Solution

Implement changelog production for Paimon C++ primary-key table writes, aligned
with Java Paimon semantics and storage format.

The implementation should cover the following producer modes:

- `input`
- Preserve input row kinds and double-write input changes to changelog files.
- Attach generated changelog files to the corresponding data increment.

- `lookup`
- Look up the previous value of each primary key when necessary.
- Produce the correct `INSERT`, `UPDATE_BEFORE`, `UPDATE_AFTER`, and `DELETE`
records.
- Generate changelog files while compacting level-0 data.

- `full-compaction`
- Produce changelog files during full compaction by comparing the compacted
result with the previous table state.

The generated changelog files should:

- Use the same file schema, row-kind encoding, metadata, and directory layout as
Java Paimon.
- Be committed through changelog manifests and referenced by the snapshot.
- Be readable through streaming scans, `audit_log`, and `binlog`.
- Work correctly with supported merge engines, sequence fields, partitions, and
bucket modes.
- Either support deletion vectors consistently with Java Paimon or reject
unsupported combinations with a clear validation error.
- Be included in commit metrics, expiration, rollback, and orphan-file cleanup.

The feature can be delivered incrementally, with separate pull requests for each
changelog producer mode.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.