apache / apache/kyuubi

[Umbrella] Suppors LakeSoul integration

Open
#4,355 2 comments 0 reactions 0 assignees View on GitHub
kind:umbrella priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues) and found no similar issues.

### Describe the proposal

[LakeSoul](https://github.com/meta-soul/LakeSoul) implements incremental upserts for both row and column and allows concurrent updates. LakeSoul uses LSM-Tree like structure to support updates on hash partitioning table with primary key, and achieve very high write throughput (30MB/s/core) on cloud object store like S3 while providing optimized merge on read performance. LakeSoul scales metadata management and achieves ACID control by using PostgreSQL. LakeSoul provides tools to ingest CDC and log streams automatically in a zero-ETL style. Kyuubi could support the integration with LakeSoul.

### Task list

Suppors LakeSoul integration.

### Are you willing to submit PR?

- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [X] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start with the linked LakeSoul project and review Kyuubi's existing integration entry points; this issue names no files or tests. Clarify the supported integration scope and requirements first, then define validation for a working, tested LakeSoul integration.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.