apple / apple/foundationdb

Create an FDB-caching role

Open
#1,250 12 comments 5 reactions 2 assignees Assigned to @dongxinEric View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Currently, FoundationDB-users might need to cache certain hot key-ranges to prevent read hot-spots. This gets complicated quickly and especially in distributed environments maintaining a consistent cache is non-trivial.

I think caching should be solved within FDB instead. After some discussions, this is roughly what we would like to implement:

1. Parameterize `KeyValueStoreMemory` so that you can create a key-value store that doesn't write to disk.
1. Introduce a new `cache`-role. Nodes with this role assigned would act like storage servers, with the difference that they don't make any data durable.
1. Introduce some form of client API (in a first release maybe through `ManagementAPI`/`fdbcli`), so that an administrator can define a key-range as being cached and a corresponding replication factor. For example, an administrator might define range `/foo` to `/foo0` as cached with replication factor 8 - which would mean that 8 cache-nodes would start fetching all updates for this range and store them.
1. Make data distributor aware of caching so that it assigns key-ranges to caching nodes. This would probably work in a very similar way as it does for storage nodes.

Obviously there are some things that need to be designed properly - mostly how to warm up caches and how the client-side load balancer should work.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.