[kv] Improve point lookup concurrency with KV writes
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
`KvTablet` currently uses one read-write lock to protect both the tablet lifecycle and the mutable write pipeline.
A point lookup acquires the read lock, while `putAsLeader` acquires the write lock for the complete write path, including row merging, auto-increment handling, pre-write-buffer updates, and WAL append. As a result, point lookups and writes to the same bucket are fully serialized, even though ordinary lookups only read the committed RocksDB state and do not access the pre-write buffer.
This unnecessarily limits lookup throughput and increases lookup tail latency under mixed read/write workloads.
### Solution
Split the current lock responsibilities into two locks:
- `kvStateLock`
- Protects the RocksDB-visible state and the `KvTablet` lifecycle.
- Ordinary lookups and puts acquire its read lock.
- Flush, snapshot, and close acquire its write lock.
- `mutationLock`
- Serializes the non-thread-safe write pipeline, including row merging, auto-increment allocation, pre-write-buffer mutation, WAL append, and rollback.
- Puts are still serialized with each other.
When both locks are required, they must always be acquired in the following order:
```text
kvStateLock -> mutationLock
```
The expected concurrency behavior is:
| Operations | Concurrent |
| --- | --- |
| Point lookup and put | Yes |
| Point lookup and point lookup | Yes |
| Put and put | No |
| Point lookup and flush | No |
| Put and flush | No |
| Point lookup and close | No |
A logical flush must hold `kvStateLock` in write mode across all RocksDB writes and the corresponding `flushedLogOffset` and row-count updates. This ensures that lookups cannot observe a partially applied flush.
Ordinary point lookups should continue reading only RocksDB, preserving the existing committed-data visibility semantics. Internal reads that also inspect the pre-write buffer must additionally acquire `mutationLock`.
RocksDB close must remain protected by `kvStateLock` in write mode so that native close cannot race with point lookup JNI calls. Long-lived scan iterators must continue holding a `ResourceGuard.Lease` because they outlive a single lock acquisition.
Callbacks that may enter Replica lifecycle code must be invoked after the internal locks are released. Backpressure-triggered flush requests must also be scheduled after releasing the read lock to avoid read-to-write lock upgrading.
### Expected benefit
This allows point lookups to overlap the expensive merge, pre-write, and WAL portions of a put while preserving existing write ordering and committed-read semantics.
The main expected benefit is improved lookup throughput and tail latency for mixed read/write workloads on the same bucket. Pure-read and pure-write workloads are not expected to change significantly.
### Correctness requirements
The change must preserve the following properties:
- Ordinary lookups never expose unflushed pre-write-buffer records.
- A lookup observes either the state before a logical flush or the completed state, never an intermediate flush segment.
- Put ordering, WAL ordering, duplicate handling, and failure rollback remain serialized.
- Snapshot data and tablet metadata represent the same state.
- No RocksDB JNI operation races with native close.
- No new lock-order cycle or read-to-write lock upgrade is introduced.
- Scanner native resources are not released while an RPC is still using them.
### Anything else?
The implementation should include concurrency tests covering:
- Point lookup overlapping an in-progress put.
- Put-put serialization.
- Lookup exclusion during flush and close.
- Visibility during a segmented RocksDB flush.
- Fatal-error callbacks executing outside internal locks.
- Scanner close racing with an in-flight continuation.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating KvTablet and the putAsLeader entry point, then trace lock usage through point lookups, flush, snapshot, close, rollback, and scanner paths. Review the existing concurrency tests and add coverage for overlapping puts and lookups, flush and close exclusion, segmented flush visibility, callbacks outside locks, and scanner continuation safety. Done means the listed concurrency and committed-read guarantees hold without lock-order cycles or native-close races.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100