Log more data distribution statistics
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
There are a variety of questions regarding the behavior of data distribution that I've wanted answered but that have been difficult to determine due to a lack of metrics. Below are some various things I've wanted to know, without too much regard for the difficulty of adding a metric for each:
* How much data is being read by data distribution from each process
* Statistics about the number of shards (maybe bytes?) being moved. I think we already have some numbers that we could use to estimate this, but it seems like having counters that get incremented when relocations start and end would be easy enough and more accurate. It may also be nice to have counts by process for the sources and destinations.
* DD Queue information for each process
* A record of teams and machine teams that persists (e.g. by rolling the events)
* Some measurement of starvation at various DD priorities.
Contributor guide
Research direction
The issue is a broad request for data-distribution metrics, covering read volume, shard movement, queue information, team history, and starvation. Start by reviewing the data distribution subsystem and its existing metric instrumentation, then determine which measurements have an agreed scope. Done would require a defined, implemented set of metrics with validation that they expose the requested behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100