apache / apache/fluss

[server] Add pre-write buffer memory metrics for primary key tables

Open
#4,307 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

### Description

For primary key table writes, every upsert/delete is buffered in an in-memory
pre-write buffer (`KvPreWriteBuffer`) until the corresponding WAL records are
persisted, and only then is it flushed to RocksDB. Each KV tablet holds one such
buffer; it is plain JVM heap — a HashMap plus a linked list of entries — and grows
with the write backlog without any capacity bound. Its footprint is currently
unobservable: the only related metrics are two truncate rate meters
(`preWriteBufferTruncateAsDuplicatedPerSecond` / `preWriteBufferTruncateAsErrorPerSecond`).
When the buffers cause GC pressure or OOM, or when flushing cannot keep up with
writes, there is no metric to attribute the cause.

Expose the buffer's basic statistics as tabletserver metrics (Sum aggregation
across all KV tablets of this server):

- `preWriteBufferMemoryUsageBytes`: estimated memory usage of the pre-write
buffers, in bytes (key/value payload bytes plus a per-entry object overhead
approximation; an observability value, not an exact measurement).
- `preWriteBufferEntryCount`: number of entries currently buffered across all
pre-write buffers.

### Scope

- Add O(1) `estimatedMemoryUsageBytes()` and `entryCount()` to `KvPreWriteBuffer`,
covering the key/value payload bytes plus a per-entry object overhead
approximation.
- Expose the buffer stats from `KvTablet`, read under the kv read lock.
- Add `preWriteBufferMemoryUsageBytes` and `preWriteBufferEntryCount` to
`MetricNames`.
- Register two tabletserver-level gauges in `TabletServerMetricGroup`, following
the existing server-level aggregated metrics pattern.
- Let `KvManager` aggregate the stats across all KV tablets it manages; dropped or
closed tablets leave the accounting automatically.
- Unit-test the buffer-level accounting across insert, flush, and truncate, and
verify the server-level gauge aggregation end to end.
- Document both metrics in the monitoring metrics reference.
- Keep the metric an explicit observability approximation, not an exact
measurement or a memory bound.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with KvPreWriteBuffer, KvTablet, KvManager, MetricNames, and TabletServerMetricGroup, following the existing server-level aggregated metrics pattern. Trace how insert, flush, truncate, and tablet removal affect the buffer state, then run the relevant unit tests and add end-to-end aggregation coverage. Done means both gauges are registered, documented in the monitoring metrics reference, and report the specified approximation across active KV tablets.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.