pingcap / pingcap/tidb

Improve Slow Log Performance and Observability

Open
#65,046 1 comment 0 reactions 0 assignees View on GitHub
component/observability type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

### Summary
Enhance the performance and observability of the TiDB slow log subsystem, including formatting, rule matching, field collection, and metrics exposure.

### Motivation
In high-concurrency or complex rule configurations, slow log formatting, rule matching, and field collection can introduce noticeable CPU and memory overhead.
At the same time, the lack of key metrics makes it difficult to evaluate the internal overhead of the slow log system itself, as well as understand dropped or skipped slow logs.

### Goals
- Reduce CPU and memory overhead in the slow log pipeline to avoid impacting normal SQL execution.
- Improve the observability of the slow log system to support online monitoring, diagnosis, and tuning.

### Tasks
- **Optimize slow log formatting**
- Currently, each slow query triggers string concatenation and formatting.
- This introduces significant overhead under high concurrency.
- **Improve metrics and observability**
- Add missing key metrics, such as:
- Slow log write rate
- Rule matching statistics
- Dropped or skipped slow log counts
- These metrics will help evaluate system overhead and diagnose slow log behavior.
- **...**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.