crypto-org-chain / crypto-org-chain/cronos
Post-rollback: mempool checkState desync across RPC nodes + stalled tx inclusion on restarted chain
- Dominant language
- Go
- Stars
- 336
- Forks
- 299
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 4
Description
### Describe the bug
Since the 2026-08-30 rollback and restart (fork at block ~90,896,189, ~11k blocks
wiped, node fork v1.7.8), two issues persist on Cronos EVM mainnet:
**1. Node `checkState` sequence is desynced across RPC/validator nodes.**
Accounts that had transactions in-flight across the rollback have a committed
sequence of N, but roughly half the public RPC nodes report
`eth_getTransactionCount(addr, "pending")` = N+1 (they still hold an uncommitted
tx in CheckTx `checkState` that will never be included), while the other half
report a clean `pending` = N. The stale nodes then reject any broadcast at nonce
N with `invalid nonce; got N, expected N+1: invalid sequence` from
`ethermint/ante.CheckAndSetEthSenderNonce` (app/ante, eth.go:318), while the
clean nodes accept the same tx but it never confirms. This leaves affected
accounts unable to submit *or* replace the stuck tx from any client — there is no
client-side path to resolve it (replacement, cancel, gas bump, different
wallet/RPC all fail). Some phantom txs eventually clear via mempool TTL; others
have persisted 3+ days.
**2. Network-wide transaction inclusion is stalled.**
Blocks are produced (~0.4–2.5/s) but ~95% are empty. Network throughput is
~5–13 tx/min (vs. normal hundreds/min). CometBFT
`rpc.cronos.org/num_unconfirmed_txs` sits stable at ~30–35 and does **not drain**
— the proposer is not including queued txs. This is not congestion.
These two compound each other: even a broadcast that passes the ante on a clean
node then sits indefinitely because inclusion is stalled.
### To Reproduce
1. Take any account that had a pending tx at the moment of the halt / within the
rolled-back window.
2. After restart, query `eth_getTransactionCount(addr, "latest")` vs `"pending"`
across multiple public RPC endpoints — the `pending` values disagree.
3. Broadcast a new tx at the committed nonce. Stale nodes reject with
`invalid sequence`; clean nodes accept but the tx is never mined.
4. Attempt a replacement / higher gas / 0-value self-tx / different wallet — same
outcome.
### Observed node split (re-verified 2026-09-02)
| RPC | `pending` vs committed |
|---|---|
| cronos.drpc.org | clean |
| cronos-evm-rpc.publicnode.com | clean |
| cronos-evm.publicnode.com | clean |
| evm.cronos.org | +1 (phantom) |
| evm-cronos.crypto.org | +1 (phantom) |
| 1rpc.io/cro | +1 (phantom) |
| rpc.vvs.finance | +1 (phantom) |
| *.rpc.thirdweb.com | +1 (phantom) |
The phantom appears to come from each restarted node replaying its cached
mempool on recheck and inflating the CheckTx `sequence`, without a corresponding
committed tx.
### Expected behavior
- Every node's CheckTx `checkState` sequence matches the committed account
sequence, so `ante.CheckAndSetEthSenderNonce` accepts a tx at the real next
nonce.
- Uncommitted txs from the rolled-back window are evicted from all node mempools.
- The proposer resumes normal transaction inclusion.
### Additional context
Possibly related, possibly separate: LayerZero EndpointV2
`0x3a73033c0b1407574c76bdbac67f126f6b4a9aa9` has emitted **zero** `PacketSent`
events for 15+ hours at a time since the restart — no OApp traffic on Cronos at
all — while the plain CRO bridge `0x6b1b50c2223eb31E0d4683b046ea9C6CB0D0ea4F`
continues to process sends normally.
### Environment
- Cronos EVM mainnet, post-rollback chain (fork v1.7.8), chainId 25 / 0x19
- RPCs checked: cronos.drpc.org, cronos-evm-rpc.publicnode.com,
cronos-evm.publicnode.com, evm.cronos.org, evm-cronos.crypto.org, 1rpc.io/cro,
rpc.vvs.finance, 25.rpc.thirdweb.com, cronos.rpc.thirdweb.com
### Suggested action
A coordinated mempool flush / forced recheck across validator and public RPC
nodes so `checkState` resyncs to committed state, plus confirmation of what
restored normal block-fullness / tx inclusion.
Contributor guide
Research direction
Start at app/ante/eth.go:318 and reproduce the pending-versus-latest nonce discrepancy across the listed RPC endpoints, then inspect CometBFT's num_unconfirmed_txs behavior after the v1.7.8 rollback. Done means the nodes agree with committed state, rolled-back transactions are no longer treated as pending, and queued transactions resume inclusion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- blockchain, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100