[kv] Batch old-value lookups in the KV write path
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
When processing a KV write batch, `KvTablet` may need to read the existing value of each key to generate the correct merged value and CDC records.
Currently, the write path checks the pre-write buffer first and then performs an individual RocksDB point lookup for every missing key. For a batch containing N distinct keys whose values are stored in RocksDB, this can result in N RocksDB JNI calls.
These lookups are executed while the tablet write lock is held. The repeated JNI crossings and serialized point lookups therefore increase the lock-holding time and limit write throughput, especially for large partial-update batches and tables using the FULL changelog image.
### Solution
atch the RocksDB old-value lookups performed while processing one KV write batch:
- Decode the incoming records and identify the keys that require an old value.
- Check the pre-write buffer first because it contains values newer than RocksDB.
- Preserve pre-write tombstones without falling back to RocksDB.
- Deduplicate the remaining keys and load them with one RocksDB `multiGet`.
- Continue processing records in their original order.
- Recheck the pre-write buffer before using a prefetched RocksDB value so that repeated keys in the same batch observe earlier mutations from that batch.
- Keep the existing fast path for WAL changelog writes that do not require old values.
This optimization should reduce RocksDB JNI crossings for old-value lookup from up to N point-get calls to at most one multi-get call per KV record batch.
The change must not alter:
- KV tablet locking semantics.
- Pre-write buffer visibility.
- Record processing order.
- WAL and CDC ordering.
- Partial-update, aggregation, versioned, first-row, or auto-increment semantics.
- Duplicate batch detection and rollback behavior.
- Flush, snapshot, recovery, or lifecycle behavior.
### Anything else?
_No response_
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the KvTablet KV write path and trace how old values are read while the tablet write lock is held. Verify the pre-write buffer, tombstone handling, repeated-key ordering, and RocksDB multiGet flow against the existing WAL and CDC behavior. Done means one batched lookup is used where applicable without changing the listed write, rollback, recovery, or lifecycle semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100