apple / apple/foundationdb

Add ability to record a log of AsyncFileCached page accesses for some time window

Open
#1,552 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

In support of https://github.com/apple/foundationdb/issues/1518, it would be nice to have an ability to log storage engine page accesses from a live cluster for use in evaluating different caching policies.

One easy to use form of this would be for the fdbcli "profile" command to support this via a subcommand. Each process will would accumulate page numbers accessed for the specified amount of time into memory and then write the collected data to a set of keys AFTER the profile period is complete (so as to not affect the workload during the collection).

The data could be logged to chunked key/value pairs with keys like `/filename/`.

Contributor guide

Open the contributing guide

Research direction

Start with the fdbcli "profile" command and the AsyncFileCached storage-engine page-access path. Determine how a live cluster could collect page numbers for a specified window and write chunked key/value records only after collection ends. Done means the profile interface supports this workflow without affecting the workload during the collection period.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.