apache / apache/paimon

[Feature] Add Databend integration to the Apache Paimon ecosystem documentation

Open
#8,700 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched the existing issues and found nothing similar.

### Motivation

Databend is an open-source analytical database built in Rust. It is currently adding native support for Paimon catalogs and tables through Apache Paimon's Rust implementation, `paimon-rust`.

Apache Paimon already provides mature integrations with Flink and Spark. The development of `paimon-rust` creates an additional opportunity to bring Paimon into Rust-native query engines and data infrastructure.

Databend can provide a real adoption and validation environment for this direction:

- Paimon provides Databend with an open table-format foundation for realtime lakehouse workloads.
- Databend provides Paimon with Rust-native distributed analytical and write workloads.
- Databend can provide `paimon-rust` with compatibility feedback from real distributed workloads.
- The integration can encourage Paimon adoption and interoperability across the Rust and Arrow data infrastructure ecosystems.
- Users gain more open and interoperable analytical engine choices for their lakehouse architectures.

The proposed Databend integration currently focuses on:

- Native integration based on Apache `paimon-rust`
- Filesystem and REST catalogs
- Distributed table scans
- Paimon system tables
- Distributed writes for supported append and primary-key tables

Databend implementation:

https://github.com/databendlabs/databend/pull/20132

### Solution

Once the Databend implementation is stable and its supported version and capability boundaries are clear, we plan to add a Databend page to Apache Paimon's ecosystem documentation.

Proposed file:

`docs/docs/ecosystem/databend.md`

The page would cover:

- Supported Databend and Paimon/`paimon-rust` versions
- Filesystem and REST Catalog configuration
- Verified read, write, and system-table capabilities
- SQL examples
- Verified data-type compatibility
- Compatibility notes where necessary
- Links to the official Databend documentation and implementation

The Databend community also plans to add corresponding Paimon documentation. The two pages will link to each other and provide users with consistent, verifiable integration information.

The documented capabilities and compatibility boundaries will be based on the final merged and tested Databend implementation.

### Anything else?

This proposal aims not only to add a documentation entry, but also to establish a public collaboration point for the two communities around:

- Compatibility feedback from real `paimon-rust` workloads
- Distributed reads and writes
- Primary-key table interoperability
- REST Catalog interoperability
- The Rust and Arrow data ecosystems
- Joint examples and lakehouse best practices

Apache Paimon provides the open table-format foundation for realtime lakehouses, `paimon-rust` brings that foundation into the Rust data ecosystem, and Databend applies it to Rust-native distributed analytical and write workloads.

Databend tracking issue:

https://github.com/databendlabs/databend/issues/20162

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with docs/docs/ecosystem/ and the Databend implementation at databendlabs/databend#20132, then check tracking issue #20162 for the final supported versions and capability boundaries. Done means adding docs/docs/ecosystem/databend.md with verified catalog configuration, read/write and system-table capabilities, SQL examples, data-type compatibility notes, and links to the official documentation and implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.