[Feature] Introduce internal log system for Paimon
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
Currently Paimon supports external log system such as kafka to reach millisecond latency for data. There may be a series of issues with using external systems, including deployment, operation and maintenance, data TTL, etc.
After a short discussion with @JingsongLi , we believe it is necessary to introduce an internal log system. It combines Paymon table format to provide the simplest subscription capability and supports millisecond level latency.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, entry points, or concrete solution are named. Start by reviewing how Paimon currently uses external Kafka log systems and its table format, then define the internal log design, subscription behavior, and millisecond-latency acceptance criteria before implementation can be scoped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100