apache / apache/fluss

Introduce a shared memory cache for PrewriteBuffer between different replica

Open
#1,950 0 comments 0 reactions 1 assignee Claimed by @swuferhong View on GitHub
component=kv
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

When testing the stability of `PrimaryKey tables`, a frequent issue arises where exceptionally high write traffic peaks cause delayed log synchronization. This leads to continuous accumulation in the KV's `PrewriteBuffer`. Since the current `PrewriteBuffer` lacks a back-pressure mechanism, has no memory cap, and utilizes JVM heap memory, it can easily trigger OOM errors, resulting in `TabletServer` crashes.

Therefore, Fluss needs to implement a back-pressure mechanism to prevent this scenario. A promising approach under consideration is to allocate a fixed-size block of off-heap memory shared across different replicas, while also imposing per-replica memory limits.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.