apache / apache/paimon

[Feature] Add Vortex columnar file format for data files

Open
#6,063 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

[Vortex](https://github.com/vortex-data/vortex) looks very promising for high-performance data processing. It offers faster queries, better compression, and lower storage costs compared to existing formats. This makes it an attractive option for users wanting to improve their analytics performance while reducing infrastructure expenses

### Solution

Use vortex as an option for data storage files in addition to already existing: orc, parquet and avro

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing how the existing ORC, Parquet, and Avro storage-file options are integrated, then read the Vortex project linked in the issue. Identify the project entry points and tests governing data-file formats; done means Vortex is available as a storage option with coverage for its integration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.