apache / apache/paimon-rust

Tracking issues of 0.1.0 version for Apache Paimon Rust

Open
#3 21 comments 21 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
197
Forks
92
Avg merge
1d 16h
Merged PRs (30d)
91

Description

Hello everyone, this is the tracking update for the paimon rust 0.1.0 release.

## Goal

Before we outline the tasks for the 0.1.0 release, let me clarify the project's goal:

**Developing a complete implementation of Paimon in pure Rust.**

- Users can read/write paimon table like they do in java API.
- Users can read/write paimon table in arrow format.
- Native support for DataFusion
- But also enable users to implement their own query engines based on this project.
- Native WASM support (a.k.a `paimon-wasm`)
- Native Python binding based on rust core (a.k.a `paimon-py`)
- Hive catalog support

## Tasks

This will be our initial release, and I aim to include basic read support in it.

- [Spec](https://paimon.apache.org/docs/master/concepts/specification/): Implement types that needed by paimon.
- [x] https://github.com/apache/paimon-rust/pull/5
- [x] Datatypes
- [x] https://github.com/apache/paimon-rust/issues/6
- [x] https://github.com/apache/paimon-rust/issues/14
- [x] https://github.com/apache/paimon-rust/issues/7
- [x] https://github.com/apache/paimon-rust/issues/10
- [ ] Global Index
- [ ] Data File Index
- Catalog
- [x] https://github.com/apache/paimon-rust/pull/62
- [x] https://github.com/apache/paimon-rust/issues/70 (leave hive catalog in next release)
- Arrow Integration:
- [x] https://github.com/apache/paimon-rust/issues/71
- Basic Read Process: No schema evolution, no merging, no deletion.
- [x] https://github.com/apache/paimon-rust/issues/72 (filter & project push down are not required in v0.1)
- IO: Integrate with [Apache OpenDAL](https://github.com/apache/opendal) for IO.
- Release Utils: Impelment scripts to help generate and verify ASF releases.

After all those tasks, I expect users can read an existing paimon table from storage services.

Contributor guide

Open the contributing guide

Research direction

Review the linked specification, issues 6, 7, 10, 14, 70, 71, and 72, along with the unchecked Global Index, Data File Index, IO, and Release Utils tasks. Start by identifying which task remains open and its dependencies. Done means the 0.1.0 tasks are complete and users can read an existing Paimon table from storage services.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust, wasm
Domain
data-engineering, release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.