Performance Diagnosis Enhancements

## Enhancement

TiDB users often suffer from performance diagnosis when trying to tune the cluster to make it most adaptive to their workloads. Some missing metrics in the system may block the diagnosis and make the tuning hard. To solve the problem, more details of queries and more metrics are required.

## Data to be collected

### Common Path

The common path includes the processes before a query is executed, connection handling, parsing, and optimization.

- [ ] Read packet duration
- [ ] Parse duration
- [ ] TSO duration
- [ ] Stmt retry count and duration(the failed stmt execution duration)
- [ ] Write response duration
- [ ] Fetch wait duration
- [ ] Execution duration = Query Duration - above all

### Read

The read requests are mainly processed in executors, and they can be divided into calculation executors and data source executors. TiDB reads data by get, batch-get, scan(iter, only used by internal txn), and coprocessor(unary request only). When TiD reads with some error, like lock error or network error, it’ll automatically retry, such retry time should be recorded, and all requests except the last successful one should be counted as retry duration.

#### Get

- [ ] Get KV size
- [ ] Get request duration
- [ ] Get request retry duration

#### BatchGet

- [ ] BatchGet KVs size
- [ ] BatchGet regions duration
- [ ] BatchGet single region duration
- [ ] BatchGet single region retry duration

#### Coprocessor

- [ ] Copr Response Size
- [ ] Fetch response duration
- [ ] Copr task wait duration
- [ ] Copr task duration
- [ ] Copr task retry duration

#### Scan

- [ ] Scan Size
- [ ] Scan request duration
- [ ] Scan request retry duration

#### UnionScan
- [ ] Read mem rows duration

#### Resolve lock For Read

- [ ] Resolve lock keys count
- [ ] Resolve locks duration
- [ ] Resolve lock wait duration
- [ ] Get txn status duration
- [ ] Get txn status retry duration
- [ ] Resolve pessimistic lock duration

### Write

Write requests are cached in TiDB’s memory before transactions are committed. In which pessimistic transactions will fetch pessimistic locks during the execution. And some write operations are based on the read results.

#### Pessimistic lock

- [ ] Lock keys count
- [ ] Lock keys duration
- [ ] Token wait duration per batch
- [ ] Lock single batch duration
- [ ] Lock single batch retry duration

#### Prewrite

- [ ] Prewrite mutation size
- [ ] Async commit get minCommitTS duration
- [ ] Prewrite mutations duration
- [ ] Token wait duration per batch
- [ ] Prewrite single batch duration
- [ ] Prewrite single batch retry duration

#### Non Async Commit

- [ ] Commit keys count
- [ ] Get commitTS duration
- [ ] Commit primary key batch duration
- [ ] Commit mutations duration
- [ ] Token wait duration per batch
- [ ] Commit single batch duration
- [ ] Commit single batch retry duration

#### Async Commit

- [ ] Commit keys count
- [ ] Commit mutations duration
- [ ] Token wait duration per batch
- [ ] Commit single batch duration
- [ ] Commit single batch retry duration

### Batch KV Client

#### Batch System

- [ ] RPC duration
- [ ] Batch size
- [ ] Batch wait duration
- [ ] Batch fetch duration
- [ ] Batch send latency

#### gRPC

- [ ] gRPC queue length
- [ ] gRPC queue wait duration
- [ ] gRPC reconnect count and duration

#### Region Cache

- [ ] Region cache miss event
- [ ] Region error count
- [ ] Load region(on cache miss) duration

## Data to be verified or enhanced
- [ ] Trasanction panel. The transaction stmt number, transaction size, transaction region size. It's needed to verify if these metrics are correct and necessary.
- [ ] The lock resolve ops panel. The difficult to understand the types and classify the impact on the cluster performance.

### Reference
- The tikv [part](https://github.com/tikv/tikv/issues/12362)
- The internal resources usage and impact [insight](https://github.com/pingcap/tidb/issues/33963).
- The incorrect txn region [metric](https://github.com/pingcap/tidb/issues/34279)