[Feature] Support Parquet row-group copy fast path for append-only compaction
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Compaction of append-only tables currently rewrites every data file: all rows are decoded,
passed through the writer pipeline, and re-encoded, even when the input files are perfectly
mergeable as-is. For Parquet (the default file format) this is unnecessarily expensive, because
row groups are self-contained compressed units — a set of Parquet files that share the same
schema and codec can be merged by concatenating their row groups directly and only rewriting
the file footer, skipping row decode/re-encode entirely.
In our production environment (Flink and Spark compaction jobs on append-only tables), a
row-group copy fast path reduced compaction kernel task time by ~59–68% with zero fallbacks.
Local micro benchmarks show 6.4–6.9× for narrow numeric tables and up to 24–32× for wide string tables (zstd, 8 MB row groups).
### Solution
An opt-in fast path in append-only compaction, controlled by
`append.compaction.row-group-copy.enabled` (default `false`). When enabled, each compaction
batch is checked for eligibility; if every input file qualifies, the batch is merged via
row-group copy, otherwise it transparently falls back to the traditional rewrite path. The
fast path therefore never changes behavior for ineligible tables and is safe to enable
unconditionally.
A batch is eligible only when **all** of the following hold for every input file:
- table format is Parquet, and the file carries the table's current schema (same schema id
and identical Parquet message type);
- uniform compression codec across all row groups and columns, matching the table's
configured codec;
- no deletion vectors, no row tracking / data evolution, no file index or bloom filter
configured, no extra files, no embedded index, no partial-column writes (`writeCols`);
- file source is `APPEND` or `COMPACT`;
- file is not encrypted and was not written with Parquet writer v2 (`BYTE_STREAM_SPLIT`
encoding).
Value stats of output files reuse the input files' stats when a file is copied in full, and
are recomputed from row-group metadata for partially copied files (when target-file-size
splitting cuts a file). Output row count is verified against input row count as a safety net.
Parquet-specific compatibility checks live in paimon-format (`ParquetRowGroupCopyChecker`),
keeping paimon-core free of Parquet internals.
Options:
- `append.compaction.row-group-copy.enabled` (default `false`): enable the fast path.
- `append.compaction.row-group-copy.preserve-page-index` (default `false`): preserve
ColumnIndex/OffsetIndex so page-level predicate pruning keeps working on compacted files,
at the cost of reading and rewriting page indexes during compaction.
- `append.compaction.row-group-copy.footer-read.parallelism` (default `1`): bounded
concurrent footer reads while preparing a compaction batch (hard cap 8).
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Already submitted: #9660.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the already-submitted PR #9660 and the paimon-format ParquetRowGroupCopyChecker mentioned in the issue. Trace the append-only compaction entry point and the three row-group-copy options. Done means eligible Parquet batches use the fast path, ineligible batches fall back safely, and the stated row-count and metadata behavior is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100