apache / apache/fluss

[kv] Batch old-value lookups in the KV write path

Open
#4,063 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

When processing a KV write batch, `KvTablet` may need to read the existing value of each key to generate the correct merged value and CDC records.

Currently, the write path checks the pre-write buffer first and then performs an individual RocksDB point lookup for every missing key. For a batch containing N distinct keys whose values are stored in RocksDB, this can result in N RocksDB JNI calls.

These lookups are executed while the tablet write lock is held. The repeated JNI crossings and serialized point lookups therefore increase the lock-holding time and limit write throughput, especially for large partial-update batches and tables using the FULL changelog image.

### Solution

atch the RocksDB old-value lookups performed while processing one KV write batch:

- Decode the incoming records and identify the keys that require an old value.
- Check the pre-write buffer first because it contains values newer than RocksDB.
- Preserve pre-write tombstones without falling back to RocksDB.
- Deduplicate the remaining keys and load them with one RocksDB `multiGet`.
- Continue processing records in their original order.
- Recheck the pre-write buffer before using a prefetched RocksDB value so that repeated keys in the same batch observe earlier mutations from that batch.
- Keep the existing fast path for WAL changelog writes that do not require old values.

This optimization should reduce RocksDB JNI crossings for old-value lookup from up to N point-get calls to at most one multi-get call per KV record batch.

The change must not alter:

- KV tablet locking semantics.
- Pre-write buffer visibility.
- Record processing order.
- WAL and CDC ordering.
- Partial-update, aggregation, versioned, first-row, or auto-increment semantics.
- Duplicate batch detection and rollback behavior.
- Flush, snapshot, recovery, or lifecycle behavior.

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the KvTablet KV write path and trace how old values are read while the tablet write lock is held. Verify the pre-write buffer, tombstone handling, repeated-key ordering, and RocksDB multiGet flow against the existing WAL and CDC behavior. Done means one batched lookup is used where applicable without changing the listed write, rollback, recovery, or lifecycle semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.