apache / apache/bookkeeper

BP-49: Support reading ahead in async mode

Open
#3,085 6 comments 12 reactions 1 assignee Claimed by @wuzhanpeng View on GitHub
type/proposal
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**BP**

> Follow the instructions at http://bookkeeper.apache.org/community/bookkeeper_proposals/ to create a proposal.

This is the master ticket for tracking BP-49 :

### Motivation

#### Current Design of Read-ahead

Under the current design of read-ahead, every `read-entry` request that the entry data is required to be read from main storage eventually, will force a read-ahead operation through the method `org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.fillReadAheadCache`. This method will read several entries after the current position and load them into the read-cache, among which the amount of entries is controlled by the `dbStorage_readAheadCacheBatchSize`.

In this mode, once a miss of read-cache occurs, the elapsed time of reading an entry is equivalent to the sum of the time of reading that entry plus reading several entries after the entry, because the process of read-ahead is synchronous.

Synchronous read-ahead is a simple and effective solution in scenarios where read latency is less of a concern. However, we found that when the cluster has a large number of catch-up reads, and the p99 latency cannot be ignored, synchronous read-ahead may introduce a lot of latency glitches. Therefore, we decided to introduce an asynchronous read-ahead mode to reduce the latency for the catch-up reads.

#### Proposed Approach

Instead of modifying the original synchronous read-ahead logic, we introduced an independent asynchronous read-ahead module named `ReadAheadManager`. The user can select a specific read-ahead mode through configuration parameters. The async read-ahead module will provide an interface for reading entry for upper-layer logic.

The following figure briefly illustrates why in the scenario of sequential catch-up read, asynchronous read-ahead can effectively improve the hit ratio and reduce read entry latency.

![image](https://user-images.githubusercontent.com/14341827/157391191-f05b71cc-1ce1-486b-9fc9-17f3b2369be9.png)

#### Evaluation Results

Before diving into the details, let's take a look at the performance optimizations brought by asynchronous read-ahead.

> The upgrade time is 2022/3/2 12:00

##### Hit / Miss Count of Read Cache
image

##### AVG Time of Read-Entry Op
image

##### P99 Time of Read-Entry Op
![image](https://user-images.githubusercontent.com/14341827/157385136-66931a6b-b982-4581-9696-bbca7093c96a.png)

Proposal PR - #3086

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.