Full node cannot stay in sync on mainnet — how is this supposed to work?

## The question

Our full node used to sit a steady 5–100 blocks behind the tip. It has now
degraded to the point where it drifts away continuously and only returns to the
tip when the process is restarted. We are currently restarting it on a timer to
keep it usable, which is clearly not how this is meant to work.

**What is the supported way for a non-validator full node to stay in sync on
mainnet today?** Specifically: is `timeout_commit = "5s"` still the right value
now that mainnet produces a block roughly every 0.4s?

## Why we think it cannot recover

When blocks arrive over gossip, the node is comfortably fast — 0.030s median to
execute and commit a block, about 8% of the ~0.40s block interval. Hardware is
nowhere near saturated (19% of one core out of 16, iowait 0).

When blocks do *not* arrive over gossip, consensus falls back to timeouts:

```
INF Timed out dur=4962.13 height=H round=0 step=RoundStepNewHeight # timeout_commit 5s
INF Timed out dur=3000 height=H round=0 step=RoundStepPropose # timeout_propose 3s
```

That is ~8s per height. Measured throughput while behind: **13 blocks in 120s
(0.11 blocks/s)** against a chain doing **2.36 blocks/s**. So once the node is
behind, the gap can only grow — it is arithmetically unable to close it through
the consensus reactor, no matter how long it runs.

Blocksync has no trouble at all:

```
INF Block Sync Rate blocks/s=45.3 height=93172573 max_peer_height=93172652
INF Time to switch to consensus reactor! height=93172652
```

45 blocks/s, ~17x the chain rate. But it only runs at process start, so the only
way we have found to get back to the tip is a restart.

## The other half: peers at or below our own height

We frequently end up with every connected peer at or below our own height. When
that happens `max_peer_height` tracks *us* rather than the chain, blocksync exits
believing it has caught up (we have seen it hand off 500+ blocks short), and
consensus then has nobody who can supply the next block:

```
height=H round=0 step=4 (Prevote)
proposal received: False, block parts: None
prevotes: BA{10:__________} 0/50800 = 0.00
```

The node then sits there indefinitely. As far as we can tell this is by design —
`enterPrevoteWait` only schedules `timeout_prevote` once `HasTwoThirdsAny()` is
true, so with zero votes no timeout is ever scheduled.

The check we now run externally is "is any connected peer's height greater than
mine". When the answer is no, the node is stuck. That feels like something the
node could detect itself.

## This has been reported before and never resolved

**#1843 "Cronos not catching the tip"** describes exactly this, including the
trigger:

> After the chain started to produce blocks faster (0.5/s) our nodes are falling
> behind ... node is getting blocks approximately 1/s therefore falling more
> behind the tip.

Same shape as ours: 16-core box, NVMe, plenty of headroom, node receiving roughly
1 block/s against a faster chain. It was closed with "Cleaning up old issues,
please let us know if it is still happening" — so, reporting back: it is still
happening, and this issue adds the measurements that were missing.

**#375** has a maintainer confirming the mechanism that makes it unrecoverable:

> the `blocksync` and `statesync` is the mechanism after the node started and
> catching to the network height. It will happen only the node starting after the
> node catches up to the network height, **will not go back to this mode**.

That matches what we see in the code (`SwitchToBlockSync` has a single caller, in
`startStateSync` at node start). So the fast path exists, works, and is
unreachable once the node is running — which is why a restart is the only remedy
we have found.

In the same thread a maintainer points at tendermint/tendermint#7141: the
`catching_up` flag is only meaningful in block sync mode. That matches what we
see — our node reports `catching_up: false` while sitting 2000 blocks behind,
which makes the obvious health check useless for detecting this state.

## Environment

**Software**

```
cronosd v1.7.8 (commit 1f072c0), built with go1.25.0
CometBFT 0.38.13
protocol p2p 8, block 11, app 0
chain cronosmainnet_25-1
role non-validator full node, voting power 0, serves public EVM JSON-RPC
OS Ubuntu 24.04.4 LTS, kernel 6.8.0-134
```

**Hardware** (VMware guest)

```
CPU AMD EPYC 7A23, 16 vCPU
RAM 31 GB (25 GB in page cache)
disk NVMe Gen4 (Kingston KC3000), 3.7 TB volume, 3% used
```

Nowhere near saturated: 19% of one core out of 16, iowait 0.

**Data sizes**

```
application.db 30 GB blockstore.db 13 GB
tx_index.db 4.2 GB state.db 3.2 GB
snapshots 4.4 GB earliest_block_height 90850001
```

**Relevant config** (`config.toml`, everything else is stock)

```toml
[p2p]
seeds = ""
persistent_peers = ""
external_address = "" # node advertised 0.0.0.0:26656 -> never
# entered remote address books. Since fixed.
laddr = "tcp://0.0.0.0:26656"
max_num_inbound_peers = 40
max_num_outbound_peers = 20
pex = true
addr_book_strict = true

[consensus]
timeout_propose = "3s"
timeout_commit = "5s" # the value this issue is mostly about

[statesync]
enable = false # node was state-synced on 2026-09-01,
# trust_height 91079194
```

`app.toml` is stock apart from `minimum-gas-prices` and the JSON-RPC block.
`[memiavl] enable = false`, `pruning = "default"`, `min-retain-blocks = 0`.

The node sits behind NAT on a private address, so until recently it had no
inbound peers at all.

## Ruled out

- Hardware, disk, memory — numbers above
- MTU / middleboxes — raw TCP to the same peers holds 12s+; conntrack 367/262144
- Version or protocol mismatch — our `node_info` is byte-identical to a healthy
at-tip node (`0.38.13`, `p2p:8 block:11 app:0`, channels `40202122233038606100`)
- Chain fork — rollback boundary verified above
- JSON-RPC load competing with consensus — 17 connections, one internal client

## Questions

1. **How should a full node stay in sync?** Is `timeout_commit = "5s"` still
correct at ~0.4s block time? The docs still show 5s. If it should be lower
for full nodes, what value do you recommend?
2. Is there a supported way to re-enter blocksync at runtime? We only found
`SwitchToBlockSync` called from `startStateSync` at node start.
3. What peer configuration do you recommend for a full node that is not
reachable from outside? Are the three official seeds in `persistent_peers`
still the recommendation?
4. Is the "peered only with lagging nodes" state a known failure mode, and is
there a recommended way to detect or break out of it?

Happy to provide full logs, `dump_consensus_state` output, or run any diagnostic.

Start by tracing SwitchToBlockSync from startStateSync and the consensus behavior around enterPrevoteWait, then compare those paths with the reported config.toml timeouts and peer states. Reproduce the behind-tip and lagging-peer conditions from the supplied logs; done means identifying the supported recovery and peer configuration, or clearly defining the required change and its validation criteria.

Written by the indexing model from the issue text.