apache / apache/gravitino

[EPIC] Add Paimon catalog support for Gravitino

Open
#1,129 14 comments 7 reactions 1 assignee Claimed by @caican00 View on GitHub
epic
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 15h
Merged PRs (30d)
315

Description

### Describe the proposal

Gravitino supports Apache Iceberg catalog at present. Apache Paimon is a streaming data lake platform that supports high-speed data ingestion, change data tracking and efficient real-time analytics. We could build Paimon catalog to support managing Paimon metadata.

Paimon exposes [`Catalog`](https://github.com/apache/incubator-paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/catalog/Catalog.java) pluggable interface and supports several implementation of `Catalog` like [`FileSystemCatalog`](https://github.com/apache/incubator-paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/catalog/FileSystemCatalog.java), [`HiveCatalog`](https://github.com/apache/incubator-paimon/blob/master/paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java). It's recommended to build a Gravitino catalog that refers to the implementations of Paimon. Meanwhile, I would propose the `RESTCatalog` interface in Paimon community.

### Task list

- [x] Propose a draft design to integrate Paimon
- [Support Apache Paimon Catalog](https://docs.google.com/document/d/1Hj61hLyr3LrYF5rdGa2IhpbXR4yaIVOEWUkL3kM6Qio/edit?usp=sharing)
- [Apache Paimon Catalog Meeting Notes](https://docs.google.com/document/d/1DHpyRIPe6BiwPB3XK-xzXFUU51ssRcJQWVE4F5umEGE/edit?usp=sharing)
- [ ] https://github.com/datastrato/gravitino/issues/2738 @SteNicholas
- [ ] Support storing Paimon catalog metadata in Gravitino storage
- [ ] Support Paimon REST interfaces

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.