apache / apache/paimon

[Feature] Introduce internal log system for Paimon

Open
#1,249 3 comments 5 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

Currently Paimon supports external log system such as kafka to reach millisecond latency for data. There may be a series of issues with using external systems, including deployment, operation and maintenance, data TTL, etc.

After a short discussion with @JingsongLi , we believe it is necessary to introduce an internal log system. It combines Paymon table format to provide the simplest subscription capability and supports millisecond level latency.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, entry points, or concrete solution are named. Start by reviewing how Paimon currently uses external Kafka log systems and its table format, then define the internal log design, subscription behavior, and millisecond-latency acceptance criteria before implementation can be scoped.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.