pingcap / pingcap/tidb

Feature Request: Automated Diagnostics for BR

Open
#55,548 0 comments 0 reactions 0 assignees View on GitHub
component/br type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Motivation

To enhance the reliability and performance analysis of the BR tool, we need an automated diagnostics feature. This feature will gather key statistics during each backup/restore process and provide insights for debugging and performance comparison. **This is especially useful in periodic tasks, helping us quickly identify instances of unusually long backup/restore times.**

## Feature Description

The proposed feature will collect statistics in two main parts:

### BR Internal Statistics
Key Parameters: Capture essential parameters used during the backup/restore process.
Stage Timing: Record the time taken to reach specific milestones, such as 10% completion of the process.
RPC Request Count: Track the total number of RPC requests initiated.
Response Time: Log the total response time for all RPC requests.
Memory usage: the limitation of progress.
### TiKV Internal Statistics
Data Size per TiKV Node: Gather the data size handled by each TiKV node.
CPU Usage: Collect CPU usage data during the backup/restore, calculated via Prometheus and returned to BR through RPC.
Request Latencies: Record latencies for various requests.
These statistics will be stored in a `statsdata` file or the current `backupmeta` file.

## New Commands and Options

### New Subcommand for br debug
Introduce a new subcommand within br debug to analyze the statistics in the `statsdata` file.
The command will output a summary of the process, including key parameters, timings, and performance metrics.
### `--compare-history` Option
Add a `--compare-history` flag to compare the current process with a previous one.
This comparison will generate a local report, highlighting differences in parameters and performance data.
If no previous `statsdata` file is available, the tool will notify the user that comparison is not possible.
Expected Impact

This feature will provide two main benefits:

1. Improved Debugging: Use the br debug subcommand to diagnose issues in a specific backup/restore process quickly.
2. Performance Analysis: The --compare-history option allows users to compare current and previous operations, helping identify whether performance issues are internal or external.

## Compatibility Considerations
N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.