apple / apple/foundationdb

Implement read cache for disk queue

Open
#2,525 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

With FDB 6.2, spill-by-reference is now the default. Currently, the disk queue is still uncached, as it was previously a read-once file. In the case of a storage server failure, there won't be a high degree of locality (spatial or temporal) between storage servers catching up, so adding a cache wouldn't be beneficial in this case.

However, for multi-region clusters, all log routers are forced to stay within 5s of each other as they peek their respective log router tags. This means that there will be a high degree of temporal locality of disk queue reads when a remote region is trying to catch up. Adding a (preferably read-only) cache onto the disk queue should help greatly reduce the read amplification of spill-by-reference when catching up from remote DCs (when number of log routers is larger than number of satellites).

Contributor guide

Open the contributing guide

Research direction

Start by locating the disk queue and spill-by-reference paths described in the issue, then trace how remote-region catch-up reads queue data. Determine where a read-only cache would fit and validate that multi-region catch-up reuses disk-queue reads without changing queue semantics. The issue names no file or test, so finding the relevant implementation and existing coverage is part of the work.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.