apache / apache/incubator-pegasus

Feature(new_metrics): migrate hotspot detection to new metrics system

Open
#1,353 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

In https://github.com/apache/incubator-pegasus/pull/1351, it has been found that `detect_hotspot_test` cannot be passed:
```
start testing write hotspot data...
/__w/incubator-pegasus/incubator-pegasus/src/test/function_test/detect_hotspot_test/test_detect_hotspot.cpp:105: Failure
Value of: find_hotkey
Actual: false
Expected: true
/__w/incubator-pegasus/incubator-pegasus/src/test/function_test/detect_hotspot_test/test_detect_hotspot.cpp:138: Failure
Expected: get_result(detection_type::write_data, key_type::hotspot_dataset) doesn't generate new fatal failures in the current thread.
Actual: it does.
/__w/incubator-pegasus/incubator-pegasus/src/test/function_test/detect_hotspot_test/test_detect_hotspot.cpp:199: Failure
Expected: write_hotspot_data() doesn't generate new fatal failures in the current thread.
Actual: it does.
[ FAILED ] detect_hotspot_test.write_hotspot_data_test (1312[85](https://github.com/apache/incubator-pegasus/actions/runs/4172815277/jobs/7225467127#step:7:86) ms)
[----------] 1 test from detect_hotspot_test (1312[86](https://github.com/apache/incubator-pegasus/actions/runs/4172815277/jobs/7225467127#step:7:87) ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1312[90](https://github.com/apache/incubator-pegasus/actions/runs/4172815277/jobs/7225467127#step:7:91) ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] detect_hotspot_test.write_hotspot_data_test

1 FAILED TEST
dsn exit with code 1
```

The reason is that the hotspot detection depends on the perf-counters system. C++ Collector collects perf-counters from each replica server, and decide if there is some partition with hotspot; once there is hotspot, C++ Collector will request the primary replica (via the replica server which the replica hosts on) to detect hotkey (`RPC_DETECT_HOTKEY`). As for the primary replica, once it has been requested to detect the hot key, it will capture each written or read hash key by `capacity_unit_calculator`.

Therefore solve this problem by following steps:

1. Temporarily disable `detect_hotspot_test` in https://github.com/apache/incubator-pegasus/pull/1351;
2. support hotspot detection by new metrics system in Go Collector, which will replace C++ Collector, detecting partitions with hotspot and requesting their primary replicas for hot key detection;
3. then, re-enable `detect_hotspot_test`.

Contributor guide

Open the contributing guide

Research direction

Start with src/test/function_test/detect_hotspot_test/test_detect_hotspot.cpp and the existing C++ Collector and Go Collector hotspot paths. Trace how perf-counters lead to RPC_DETECT_HOTKEY and how capacity_unit_calculator captures keys. Done means hotspot detection is supported by the Go Collector and detect_hotspot_test can be re-enabled and passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.