Cronos mainnet archive: eth_getBlockReceipts count mismatch and tx hash->block inconsistency on historical blocks

# Cronos Mainnet Archive Node Case: `eth_getBlockReceipts` Count / Tx Mapping Inconsistency

## Summary

We are running Cronos mainnet **archive** nodes and observing a reproducible inconsistency on historical blocks:

- `eth_getBlockByNumber(, true)` returns `N` transactions
- `eth_getBlockReceipts()` returns `N-1` receipts on our node for some early blocks
- For at least one missing tx hash, `eth_getTransactionByHash` resolves to a different block on our node than on public Cronos official RPC

This ticket provides our architecture, effective configuration, concrete reproduction evidence, and asks for guidance on the correct upstream remediation path.

---

## 1) Archive Architecture

Our Cronos archive node is built with:

- **RocksDB** as base backend (`blockstore.db`, `state.db`, `tx_index.db`, `application.db`, `evmindexer.db`)
- **MemIAVL** enabled for latest-state access
- **VersionDB** enabled for historical-state queries
- **EVM indexer** enabled (`enable-indexer = true`)

Key operational characteristics:

- Archive mode (`pruning: nothing`)
- Heavy historical JSON-RPC workload (`eth_getLogs`, `eth_call` at historical heights, traces)
- `index-eth-tx backward` was used to backfill `evmindexer.db`

---

## 2) Effective Config (shareable excerpts)

### Runtime values used on affected archive node

Relevant runtime values:

```yaml
db_backend: rocksdb
tx_index_indexer: kv

versiondb:
enabled: true

memiavl:
enabled: true
asyncCommitBuffer: 8
zeroCopy: false
cacheSize: 50000
snapshotKeepRecent: 0
snapshotInterval: 1000

enableIndexer: true
allowIndexerGap: true

pruning: nothing

image_tag: v1.7.5
```

### Rendered `app.toml` (relevant sections)

```toml
[json-rpc]
enable = true
api = "eth,net,web3,txpool,debug"
enable-indexer = true
allow-indexer-gap = true

[versiondb]
enable = true

[memiavl]
async-commit-buffer = 8
enable = true
```

### Rendered `config.toml` (relevant sections)

```toml
db_backend = "rocksdb"

[tx_index]
indexer = "kv"
```

---

## 3) Problem Description and Evidence

## Observed behavior

Example block: `0xb1190c`

On our archive node:

```bash
curl -s -X POST http://localhost:8545 \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0xb1190c",true],"id":1}' \
| jq '{block: .result.number, tx_count: (.result.transactions | length)}'
# => tx_count: 1443

curl -s -X POST http://localhost:8545 \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_getBlockReceipts","params":["0xb1190c"],"id":1}' \
| jq '{block: "0xb1190c", receipt_count: (.result | length)}'
# => receipt_count: 1442
```

Observed output on localhost:

```json
{
"block": "0xb1190c",
"tx_count": 1443
}
{
"block": "0xb1190c",
"receipt_count": 1442
}
```

On public Cronos official RPC (`https://evm.cronos.org`) for the same block:

```bash
curl -s -X POST https://evm.cronos.org \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0xb1190c",true],"id":1}' \
| jq '{block: .result.number, tx_count: (.result.transactions | length)}'

curl -s -X POST https://evm.cronos.org \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"eth_getBlockReceipts","params":["0xb1190c"],"id":1}' \
| jq '{block: "0xb1190c", receipt_count: (.result | length)}'
```

Observed output on public official RPC:

```json
{
"block": "0xb1190c",
"tx_count": 1443
}
{
"block": "0xb1190c",
"receipt_count": 1443
}
```

This also reproduces on another early block (`0x198e`) where local node has 1 missing receipt while official public RPC does not.

Additional reproduced block (`0x198e`):

```json
// localhost
{
"block": "0x198e",
"tx_count": 3
}
{
"block": "0x198e",
"receipt_count": 2
}

// https://evm.cronos.org
{
"block": "0x198e",
"tx_count": 3
}
{
"block": "0x198e",
"receipt_count": 3
}
```

### Missing tx hash in block receipts (local)

For block `0xb1190c`, tx present in block body but absent in `eth_getBlockReceipts` response on local node:

- `0x79eed4fe2578c791700fb63be1a8ee72fe11f0b55ac0527474b24b00a73dd94b`

### Tx hash mapping mismatch

For that same tx hash:

- On local node: `eth_getTransactionByHash` resolves to `blockNumber = 0xb118fc`
- On public Cronos official RPC: resolves to `blockNumber = 0xb1190c`

This indicates mismatch in historical tx/receipt indexing and/or lookup path on our node.

---

## 4) Why we believe this is related to known historical classes

From Cronos repo history/changelog and related threads:

- [#1823](https://github.com/crypto-org-chain/cronos/pull/1823) add support for `eth_getBlockReceipts`
- [#1991](https://github.com/crypto-org-chain/cronos/pull/1991) fix `eth_getBlockReceipts` crash
- [#526](https://github.com/crypto-org-chain/cronos/pull/526) fix Tendermint duplicated tx issue
- [#513](https://github.com/crypto-org-chain/cronos/pull/513) add `fix-unlucky-tx` command
- [#587](https://github.com/crypto-org-chain/cronos/pull/587) unlucky tx patch recompute eth tx hash
- [#549](https://github.com/crypto-org-chain/cronos/pull/549) use custom tx indexer feature of Ethermint
- [#502](https://github.com/crypto-org-chain/cronos/pull/502) failed tx handling in JSON-RPC APIs

Potentially related issue threads:

- [#521](https://github.com/crypto-org-chain/cronos/issues/521) no command to patch duplicated tx situation
- [#576](https://github.com/crypto-org-chain/cronos/issues/576) patch-unlucky-tx handling duplicated tx correctly
- [evmos/ethermint#1045](https://github.com/evmos/ethermint/issues/1045) failed tx visibility/receipt semantics
- [evmos/ethermint#781](https://github.com/evmos/ethermint/pull/781) tx indexing consistency in RPC paths

---

## 5) Code-path context we validated

From the Cronos build dependency (`github.com/crypto-org-chain/ethermint`), `eth_getBlockReceipts` is served via Ethermint backend path:

- `rpc/namespaces/ethereum/eth/api.go` -> `GetBlockReceipts`
- `rpc/backend/blocks.go` -> `GetBlockReceipts`

That path builds tx hash list from block and resolves each receipt through tx-hash lookup/indexer path.

Given our evidence (same tx hash mapped to different blocks locally vs official RPC), this appears aligned with tx-index / EVM-index historical inconsistency rather than node sync lag.

---

## 6) What we already verified

- Archive architecture is correct and enabled (`rocksdb + memiavl + versiondb + evm indexer`)
- Node is healthy for general archive RPC workloads
- Mismatch is reproducible on specific early blocks
- Public Cronos official RPC does not show same mismatch for tested sample blocks

---

## 7) Request for Cronos team

Please advise the **recommended and safe remediation path** for this exact mismatch class.

Specifically:

1. Is this a known historical data/index issue class tied to duplicated/unlucky tx eras?
2. Should remediation be:
- full `evmindexer.db` rebuild via `index-eth-tx backward`, or
- targeted `tx_index` / blockstore patching, or
- both in specific order?
3. For current builds where patch commands differ across versions, what is the canonical procedure now?
4. Are there known commit ranges / release versions that must be used to avoid this mismatch permanently?
5. Is there a validation script/checklist from Cronos team to confirm node is fully corrected after remediation?

---

## 8) Minimal repro bundle (for maintainers)

If useful, we can provide:

- block list exhibiting mismatch
- missing tx hash list per block
- local vs official RPC tx-hash-to-block comparison CSV
- node image/tag and full rendered `app.toml` / `config.toml` excerpts

---

## Environment details

- Chain: `cronosmainnet_25-1`
- Node role: archive RPC
- Image observed on affected node: `v1.7.5`

Start with the reported Ethermint paths rpc/namespaces/ethereum/eth/api.go and rpc/backend/blocks.go, especially GetBlockReceipts and its transaction lookup path. Compare the local and official results for blocks 0xb1190c and 0x198e, then inspect the tx indexer and the referenced historical patch commands. Done means the maintainers define and validate a safe remediation procedure for the affected archive data.

Written by the indexing model from the issue text.