Introduce a shared memory cache for PrewriteBuffer between different replica
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
When testing the stability of `PrimaryKey tables`, a frequent issue arises where exceptionally high write traffic peaks cause delayed log synchronization. This leads to continuous accumulation in the KV's `PrewriteBuffer`. Since the current `PrewriteBuffer` lacks a back-pressure mechanism, has no memory cap, and utilizes JVM heap memory, it can easily trigger OOM errors, resulting in `TabletServer` crashes.
Therefore, Fluss needs to implement a back-pressure mechanism to prevent this scenario. A promising approach under consideration is to allocate a fixed-size block of off-heap memory shared across different replicas, while also imposing per-replica memory limits.
### Solution
_No response_
### Anything else?
_No response_
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.