apache / apache/paimon-cpp

[Feature] Support Vortex file format

Open
#319 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
65
Forks
25
Avg merge
2d 12h
Merged PRs (30d)
80

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.

### Motivation

Apache Paimon Java supports Vortex through the paimon-vortex module, a Rust-based columnar file format with extensible compression encodings and lazily loaded summary statistics. Paimon C++ cannot currently access tables using this format.

### Solution

Add Vortex support to the Paimon C++ file-format SPI:
- Register the vortex file-format factory.
- Implement compatible schema, reader, and writer handling.
- Support column projection, predicate pushdown, and precise row selection.
- Validate the same supported and unsupported logical types as Java.
- Add C++ round-trip tests and Java-produced file compatibility tests.

### Anything else?

Vortex is implemented in Rust, so the integration can follow the Mosaic Rust FFI patterns (#238).
Java reference module: https://github.com/apache/paimon/tree/master/paimon-vortex

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reading the Java reference module at paimon-vortex and the Mosaic Rust FFI patterns in issue #238. Map the C++ file-format SPI requirements to the requested schema, reader, writer, projection, predicate, and row-selection behavior. Done means C++ round-trip tests and Java-produced file compatibility tests cover supported and unsupported logical types.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java, rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.