apache / apache/paimon

[Feature] Support Parquet row-group copy fast path for append-only compaction

Open
#9,664 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Compaction of append-only tables currently rewrites every data file: all rows are decoded,
passed through the writer pipeline, and re-encoded, even when the input files are perfectly
mergeable as-is. For Parquet (the default file format) this is unnecessarily expensive, because
row groups are self-contained compressed units — a set of Parquet files that share the same
schema and codec can be merged by concatenating their row groups directly and only rewriting
the file footer, skipping row decode/re-encode entirely.

In our production environment (Flink and Spark compaction jobs on append-only tables), a
row-group copy fast path reduced compaction kernel task time by ~59–68% with zero fallbacks.
Local micro benchmarks show 6.4–6.9× for narrow numeric tables and up to 24–32× for wide string tables (zstd, 8 MB row groups).

### Solution

An opt-in fast path in append-only compaction, controlled by
`append.compaction.row-group-copy.enabled` (default `false`). When enabled, each compaction
batch is checked for eligibility; if every input file qualifies, the batch is merged via
row-group copy, otherwise it transparently falls back to the traditional rewrite path. The
fast path therefore never changes behavior for ineligible tables and is safe to enable
unconditionally.

A batch is eligible only when **all** of the following hold for every input file:

- table format is Parquet, and the file carries the table's current schema (same schema id
and identical Parquet message type);
- uniform compression codec across all row groups and columns, matching the table's
configured codec;
- no deletion vectors, no row tracking / data evolution, no file index or bloom filter
configured, no extra files, no embedded index, no partial-column writes (`writeCols`);
- file source is `APPEND` or `COMPACT`;
- file is not encrypted and was not written with Parquet writer v2 (`BYTE_STREAM_SPLIT`
encoding).

Value stats of output files reuse the input files' stats when a file is copied in full, and
are recomputed from row-group metadata for partially copied files (when target-file-size
splitting cuts a file). Output row count is verified against input row count as a safety net.

Parquet-specific compatibility checks live in paimon-format (`ParquetRowGroupCopyChecker`),
keeping paimon-core free of Parquet internals.

Options:

- `append.compaction.row-group-copy.enabled` (default `false`): enable the fast path.
- `append.compaction.row-group-copy.preserve-page-index` (default `false`): preserve
ColumnIndex/OffsetIndex so page-level predicate pruning keeps working on compacted files,
at the cost of reading and rewriting page indexes during compaction.
- `append.compaction.row-group-copy.footer-read.parallelism` (default `1`): bounded
concurrent footer reads while preparing a compaction batch (hard cap 8).

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Already submitted: #9660.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the already-submitted PR #9660 and the paimon-format ParquetRowGroupCopyChecker mentioned in the issue. Trace the append-only compaction entry point and the three row-group-copy options. Done means eligible Parquet batches use the fast path, ineligible batches fall back safely, and the stated row-count and metadata behavior is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.