apple / apple/foundationdb

More efficient and detailed information about DD's situation and data movement decisions

Open
#2,660 2 comments 0 reactions 1 assignee Claimed by @negoyal View on GitHub
data distribution
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Current data distribution (DD) logs the relocate shard request via TraceEvent into telemetry database.
Because of the volume of the data and the limited quota of the telemetry database, DD cannot log every decision and shard usage (bytes, reads, writes) into the telemetry database.

Even if the telemetry database is not limited by capacity, querying those data can be slow.

Two scenarios users can benefit from more data and more efficient query of the data:
**Scenario 1**: Developers of DD wants to improve DD. They need to understand how the current DD performs, the per-shard usage in history, how the decision is made and how the per-shard usage is changed because of a particular decision; This requires logging all instead of sampled shard info and querying the data quickly;

**Scenario 2**: Clients of FDB found the cluster's performance degraded due to DD issue. Clients can be throttled by the near-future FDB when DD struggles to balance the capacity or traffic (hot shard). Developers of clients may want to know what the specific problem DD has and can change how clients use FDB. However, clients may not have permission to access the telemetry database that holds the trace events of the entire FDB luster. Saving the selected detailed telemetry data can help clients identify the problem.

We can potentially save the per-shard usage (bytes, reads, writes) and relocate shard request into a dedicated system key space on a rolling basis. To prevent the visibility data from taking too much space, we can limit the size of the special key space and retain only a short period of data by override old data.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.