Enable the profiling, goroutines, metrics endpoints and signal handler as soon as possible
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
For now, during the TiDB boot process, the observability infrastructures are not started, which makes it pretty hard for us to investigate any issue before starting the server. I think the following ones are especially important and helpful for debug:
1. `/debug/pprof`, especially `/debug/pprof/profile` and `/debug/pprof/goroutine`. If the TiDB is blocked in the boot stage, get the backtrace of all goroutines and help us to understand where it is blocked.
2. `/metrics`. Without it, the prometheus cannot scrape the metrics of the booting TiDB server.
3. `SIGUSR1` signal handler. After starting the server, the TiDB server will handle `SIGUSR1` signal by printing the backtrace of all goroutines. Without it, all other signals which will print the backtrace like `SIGQUIT` will kill the process.
## Current behavior
All of them are started after creating `storage`, `dom`, `server`, etc.
```go
storage, dom := createStoreDDLOwnerMgrAndDomain(keyspaceName)
repository.SetupRepository(dom)
svr := createServer(storage, dom)
exited := make(chan struct{})
signal.SetupSignalHandler(func() {
svr.Close()
resourcemanager.InstanceResourceManager.Stop()
cleanup(svr, storage, dom)
cpuprofile.StopCPUProfiler()
executor.Stop()
close(exited)
})
topsql.SetupTopSQL(svr)
terror.MustNil(svr.Run(dom))
```
The signal handler is setup in `signal.SetupSignalHandler`. The status server is setup in `svr.Run(dom)`.
## Change
To overcome this inconvenience, I propose to make the following changes:
1. Create a temporary status server including metrics and `/debug/pprof` before `svr.Run(dom)`, and stop it before creating a new fully functional status server.
2. Setup the signal handler of `SIGUSR1` earlier.
Contributor guide
Assessment
This issue has not been assessed yet.