apache / apache/incubator-pegasus
Feature(new_metrics): migrate hotspot detection to new metrics system
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
In https://github.com/apache/incubator-pegasus/pull/1351, it has been found that `detect_hotspot_test` cannot be passed:
```
start testing write hotspot data...
/__w/incubator-pegasus/incubator-pegasus/src/test/function_test/detect_hotspot_test/test_detect_hotspot.cpp:105: Failure
Value of: find_hotkey
Actual: false
Expected: true
/__w/incubator-pegasus/incubator-pegasus/src/test/function_test/detect_hotspot_test/test_detect_hotspot.cpp:138: Failure
Expected: get_result(detection_type::write_data, key_type::hotspot_dataset) doesn't generate new fatal failures in the current thread.
Actual: it does.
/__w/incubator-pegasus/incubator-pegasus/src/test/function_test/detect_hotspot_test/test_detect_hotspot.cpp:199: Failure
Expected: write_hotspot_data() doesn't generate new fatal failures in the current thread.
Actual: it does.
[ FAILED ] detect_hotspot_test.write_hotspot_data_test (1312[85](https://github.com/apache/incubator-pegasus/actions/runs/4172815277/jobs/7225467127#step:7:86) ms)
[----------] 1 test from detect_hotspot_test (1312[86](https://github.com/apache/incubator-pegasus/actions/runs/4172815277/jobs/7225467127#step:7:87) ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1312[90](https://github.com/apache/incubator-pegasus/actions/runs/4172815277/jobs/7225467127#step:7:91) ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] detect_hotspot_test.write_hotspot_data_test
1 FAILED TEST
dsn exit with code 1
```
The reason is that the hotspot detection depends on the perf-counters system. C++ Collector collects perf-counters from each replica server, and decide if there is some partition with hotspot; once there is hotspot, C++ Collector will request the primary replica (via the replica server which the replica hosts on) to detect hotkey (`RPC_DETECT_HOTKEY`). As for the primary replica, once it has been requested to detect the hot key, it will capture each written or read hash key by `capacity_unit_calculator`.
Therefore solve this problem by following steps:
1. Temporarily disable `detect_hotspot_test` in https://github.com/apache/incubator-pegasus/pull/1351;
2. support hotspot detection by new metrics system in Go Collector, which will replace C++ Collector, detecting partitions with hotspot and requesting their primary replicas for hot key detection;
3. then, re-enable `detect_hotspot_test`.
Contributor guide
Research direction
Start with src/test/function_test/detect_hotspot_test/test_detect_hotspot.cpp and the existing C++ Collector and Go Collector hotspot paths. Trace how perf-counters lead to RPC_DETECT_HOTKEY and how capacity_unit_calculator captures keys. Done means hotspot detection is supported by the Go Collector and detect_hotspot_test can be re-enabled and passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, go
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100