iotexproject / iotexproject/iotex-core

startup indexer catch-up is unobservable: admin/pprof server only starts after svr.Start() returns

Open
#4,965 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.6k
Forks
382
Avg merge
4d 22h
Merged PRs (30d)
17

Description

## Problem

`StartServer` brings the admin mux up **after** `svr.Start()` returns:

```go
// server/itx/server.go
func StartServer(ctx context.Context, svr *Server, probeSvr *probe.Server, cfg config.Config) {
if err := svr.Start(ctx); err != nil { ... } // <- entire startup catch-up happens in here
...
var adminserv http.Server
if cfg.System.HTTPAdminPort > 0 {
// pprof, /ha, /pause, /producer-keys, log level control
}
<-ctx.Done()
```

But `svr.Start()` runs the startup indexer catch-up inline: `blockdao.Start` → `checkIndexers` → `blockIndexerChecker.CheckIndexer`, which replays every block between the state indexer height and the block DAO tip. On a node that has been offline for a while, or is restoring from a snapshot well behind the tip, that is hours to days of work.

For the whole of that phase the node is **completely unobservable**: no pprof, no `/debug/pprof/heap`, no runtime log-level control. The probe server is up, but it only serves `/liveness`, `/readiness`, `/health` and `/metrics`.

The one phase where profiling is most likely to be needed is the one phase where it is unavailable.

## Impact

Concretely: diagnosing #4964 (from-genesis replay degrading quadratically) was blocked on this. `wget http://127.0.0.1:/debug/pprof/profile` inside the container returned `Connection refused` for as long as the replay ran. It took a patched build to get the CPU and heap profiles that identified the hot path.

Any operator hitting a slow or stuck catch-up today has no way to see what the node is doing, short of rebuilding it.

## Suggested fix

Move the admin mux and its listener ahead of `svr.Start()`. It binds to `127.0.0.1` only, so exposure is unchanged.

One caveat worth handling in the same change: the goroutine that starts the listener also does

```go
runtime.SetMutexProfileFraction(1)
runtime.SetBlockProfileRate(1)
```

Rate-1 block profiling records a stack trace for every blocking event. Moving that ahead of `svr.Start()` would put that overhead on the catch-up, which is the CPU-bound phase. **CPU and heap profiles do not need either setting** — only `/debug/pprof/block` and `/mutex` do. So the mux and listener should move, while those two calls stay where they are (or become opt-in).

A working implementation of the move, with the two `runtime.Set*` calls left behind as described above, is [in a comment below](https://github.com/iotexproject/iotex-core/issues/4965#issuecomment-5299166169). It is the build that was used to profile the 39M-52M replay segment.

## Note

Filed as a follow-up rather than a PR at the maintainers' preference; happy to raise one if wanted.

Contributor guide

Open the contributing guide

Research direction

Start in server/itx/server.go at StartServer and trace svr.Start through the startup indexer catch-up. Move the admin mux and listener early enough for pprof and admin endpoints to respond during catch-up, while keeping the runtime.SetMutexProfileFraction and SetBlockProfileRate calls out of that CPU-bound phase. Done means the admin server is reachable before svr.Start() returns without changing its loopback-only exposure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.