apache / apache/fluss

[server] add the metric of KV preWriteBuffer

Open
#3,532 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

Under high write throughput on a single tablet server, if replication falls behind, the KV preWriterBuffer may be unable to release buffered data because the high watermark has not advanced, eventually leading to OOM on the tablet server.

for the above case, we should do the following ways to eliminate risk
1. introduce the metric to observe
2. introduce the backpressure mechanism to limit client writing speed.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the tablet-server implementation and references to the KV preWriterBuffer, then trace how buffered data and the high watermark are handled during replication lag. The issue is done when the buffer can be observed through a metric and the proposed client-write backpressure behavior is defined, implemented, and covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.