iotexproject / iotexproject/iotex-core
startup indexer catch-up is unobservable: admin/pprof server only starts after svr.Start() returns
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 382
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 17
Description
## Problem
`StartServer` brings the admin mux up **after** `svr.Start()` returns:
```go
// server/itx/server.go
func StartServer(ctx context.Context, svr *Server, probeSvr *probe.Server, cfg config.Config) {
if err := svr.Start(ctx); err != nil { ... } // <- entire startup catch-up happens in here
...
var adminserv http.Server
if cfg.System.HTTPAdminPort > 0 {
// pprof, /ha, /pause, /producer-keys, log level control
}
<-ctx.Done()
```
But `svr.Start()` runs the startup indexer catch-up inline: `blockdao.Start` → `checkIndexers` → `blockIndexerChecker.CheckIndexer`, which replays every block between the state indexer height and the block DAO tip. On a node that has been offline for a while, or is restoring from a snapshot well behind the tip, that is hours to days of work.
For the whole of that phase the node is **completely unobservable**: no pprof, no `/debug/pprof/heap`, no runtime log-level control. The probe server is up, but it only serves `/liveness`, `/readiness`, `/health` and `/metrics`.
The one phase where profiling is most likely to be needed is the one phase where it is unavailable.
## Impact
Concretely: diagnosing #4964 (from-genesis replay degrading quadratically) was blocked on this. `wget http://127.0.0.1:/debug/pprof/profile` inside the container returned `Connection refused` for as long as the replay ran. It took a patched build to get the CPU and heap profiles that identified the hot path.
Any operator hitting a slow or stuck catch-up today has no way to see what the node is doing, short of rebuilding it.
## Suggested fix
Move the admin mux and its listener ahead of `svr.Start()`. It binds to `127.0.0.1` only, so exposure is unchanged.
One caveat worth handling in the same change: the goroutine that starts the listener also does
```go
runtime.SetMutexProfileFraction(1)
runtime.SetBlockProfileRate(1)
```
Rate-1 block profiling records a stack trace for every blocking event. Moving that ahead of `svr.Start()` would put that overhead on the catch-up, which is the CPU-bound phase. **CPU and heap profiles do not need either setting** — only `/debug/pprof/block` and `/mutex` do. So the mux and listener should move, while those two calls stay where they are (or become opt-in).
A working implementation of the move, with the two `runtime.Set*` calls left behind as described above, is [in a comment below](https://github.com/iotexproject/iotex-core/issues/4965#issuecomment-5299166169). It is the build that was used to profile the 39M-52M replay segment.
## Note
Filed as a follow-up rather than a PR at the maintainers' preference; happy to raise one if wanted.
Contributor guide
Research direction
Start in server/itx/server.go at StartServer and trace svr.Start through the startup indexer catch-up. Move the admin mux and listener early enough for pprof and admin endpoints to respond during catch-up, while keeping the runtime.SetMutexProfileFraction and SetBlockProfileRate calls out of that CPU-bound phase. Done means the admin server is reachable before svr.Start() returns without changing its loopback-only exposure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100