pingcap / pingcap/tidb

Enable the profiling, goroutines, metrics endpoints and signal handler as soon as possible

Open
#59,087 3 comments 6 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

For now, during the TiDB boot process, the observability infrastructures are not started, which makes it pretty hard for us to investigate any issue before starting the server. I think the following ones are especially important and helpful for debug:

1. `/debug/pprof`, especially `/debug/pprof/profile` and `/debug/pprof/goroutine`. If the TiDB is blocked in the boot stage, get the backtrace of all goroutines and help us to understand where it is blocked.
2. `/metrics`. Without it, the prometheus cannot scrape the metrics of the booting TiDB server.
3. `SIGUSR1` signal handler. After starting the server, the TiDB server will handle `SIGUSR1` signal by printing the backtrace of all goroutines. Without it, all other signals which will print the backtrace like `SIGQUIT` will kill the process.

## Current behavior

All of them are started after creating `storage`, `dom`, `server`, etc.

```go
storage, dom := createStoreDDLOwnerMgrAndDomain(keyspaceName)
repository.SetupRepository(dom)
svr := createServer(storage, dom)

exited := make(chan struct{})
signal.SetupSignalHandler(func() {
svr.Close()
resourcemanager.InstanceResourceManager.Stop()
cleanup(svr, storage, dom)
cpuprofile.StopCPUProfiler()
executor.Stop()
close(exited)
})
topsql.SetupTopSQL(svr)
terror.MustNil(svr.Run(dom))
```

The signal handler is setup in `signal.SetupSignalHandler`. The status server is setup in `svr.Run(dom)`.

## Change

To overcome this inconvenience, I propose to make the following changes:

1. Create a temporary status server including metrics and `/debug/pprof` before `svr.Run(dom)`, and stop it before creating a new fully functional status server.
2. Setup the signal handler of `SIGUSR1` earlier.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.