[Feature] Support changelog production
- Vorherrschende Sprache
- C++
- Sterne
- 65
- Forks
- 25
- Ø Merge
- 2 T. 12 Std.
- Gemergte PRs (30 T.)
- 80
Beschreibung
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.
### Motivation
Java Paimon supports producing changelog files for primary-key tables through the
`changelog-producer` option, including `input`, `lookup`, and `full-compaction`
modes.
Paimon C++ already contains part of the required infrastructure, such as:
- Parsing the `changelog-producer` option.
- Changelog file metadata in `DataIncrement` and `CompactIncrement`.
- Changelog manifest and snapshot commit support.
- Lookup changelog merge functions and changelog compaction rewriters.
- Streaming changelog scans and the `audit_log` and `binlog` system tables.
However, schema validation currently rejects every value except `none`, and the
writer does not produce changelog files. As a result, primary-key tables written
by Paimon C++ cannot provide the same changelog behavior as tables written by
Java Paimon.
Supporting Java-compatible changelog production would enable incremental
downstream consumption and improve interoperability between Java Paimon and
Paimon C++.
### Solution
Implement changelog production for Paimon C++ primary-key table writes, aligned
with Java Paimon semantics and storage format.
The implementation should cover the following producer modes:
- `input`
- Preserve input row kinds and double-write input changes to changelog files.
- Attach generated changelog files to the corresponding data increment.
- `lookup`
- Look up the previous value of each primary key when necessary.
- Produce the correct `INSERT`, `UPDATE_BEFORE`, `UPDATE_AFTER`, and `DELETE`
records.
- Generate changelog files while compacting level-0 data.
- `full-compaction`
- Produce changelog files during full compaction by comparing the compacted
result with the previous table state.
The generated changelog files should:
- Use the same file schema, row-kind encoding, metadata, and directory layout as
Java Paimon.
- Be committed through changelog manifests and referenced by the snapshot.
- Be readable through streaming scans, `audit_log`, and `binlog`.
- Work correctly with supported merge engines, sequence fields, partitions, and
bucket modes.
- Either support deletion vectors consistently with Java Paimon or reject
unsupported combinations with a clear validation error.
- Be included in commit metrics, expiration, rollback, and orphan-file cleanup.
The feature can be delivered incrementally, with separate pull requests for each
changelog producer mode.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.