crypto-org-chain / crypto-org-chain/cronos

cannot query with height in the future; please provide a valid height

Open
#747 7 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
336
Forks
299
Avg merge
2d 17h
Merged PRs (30d)
4

Description

**Describe the bug**
- Cronos nodes fall out of sync
- We received alerts on different days between 14:20 - 17:30 UTC
- Then it synced again without any changes, so it looks related to load or related to something else that happens at that time of the day
- Workload was low during the issue, plenty of hardware resources left
- The same issue happened on multiple nodes at the same time
- The error that printed many times at similar times that the out of sync alert in out systems was:
`cronosd[211409]: 1:11PM INF Served eth_call conn=165.227.220.115:44408 duration=291.995272 err="rpc error: code = Unknown desc = cannot query with height in the future; please provide a valid height: invalid height" module=geth reqid=1`

**To Reproduce**
- Not sure what to do to reproduce, it seems adding a little load does it but we had to add more nodes and it helped as workaround. So at the moment it is not reproducible on our end, but we are wasting a lot of resources since the nodes were already at 5-10% usage (cpu/mem) and we had to add more.

**Expected behavior**
- Node stays synced and it doesn't throw this error

**Additional context**
- Last time seen at v0.8.0
- Current version: binary says v0.8.0 but I think we are on v0.8.1 (I did download once the v0.8.1 binary and it still threw v0.8.0 if I ran the version command)
- We don't think v0.8.1 or v0.8.2 fixes this since I do not see any reports of this error on the releases `cannot query with height in the future`
- We observed this problem on nodes that are on archive mode, we don't have non-archive to compare

**Things we have tried**
- We made contact with Anthea on Cronos discord and he suggested to check the system time but they are ok:
Local time: Tue 25 Oct 2022 07:57:56 AM UTC (24h 07:57)
Nodes time: (ran ssh host date on our nodes)
Tue Oct 25 07:57:57 UTC 2022
Tue Oct 25 07:57:57 UTC 2022
Tue Oct 25 07:57:57 UTC 2022
Tue Oct 25 07:57:57 UTC 2022
Tue Oct 25 07:57:57 UTC 2022
Tue Oct 25 07:57:57 UTC 2022

- Next suggestion is increasing open file limit
ulimit -n 2000000
ulimit -a

We are using 200.000 not 2.000.000 open file limit (ulimit -n)

Also, observed a possible correlation between uptime and # of open files, like there is some problem and cronos is not closing unused open files and they grow over time:

Running: sudo lsof | wc -l
CRONOS-ARCHIVE-1
CRONOS-ARCHIVE-2
CRONOS-ARCHIVE-3
CRONOS-ARCHIVE-4
CRONOS-TESTNET-1
CRONOS-TESTNET-2
CRONOS-TESTNET-3

Results:
683235
361592
14695
367258
135280
134014
135285

Running (on same servers): uptime
08:17:13 up 49 days, 16:04, 1 user, load average: 0.48, 0.37, 0.31
08:17:14 up 7 days, 20:42, 0 users, load average: 0.27, 0.29, 0.28
08:17:16 up 7 days, 21:41, 0 users, load average: 1.13, 1.30, 1.19
08:17:18 up 7 days, 20:14, 0 users, load average: 0.12, 0.15, 0.16
08:17:23 up 96 days, 15:55, 0 users, load average: 0.19, 0.15, 0.11
08:17:24 up 96 days, 15:00, 0 users, load average: 0.11, 0.12, 0.13
08:17:26 up 96 days, 13:27, 0 users, load average: 0.13, 0.24, 0.21

Contributor guide

Open the contributing guide

Research direction

Start with the cronosd logs containing the repeated eth_call error and compare them with node sync status, uptime, open-file counts, archive mode, and system time. The issue does not identify a source file or test, and reproduction is unavailable; done means identifying and fixing the cause so nodes remain synced without future-height errors under the reported conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
blockchain, go
Domain
blockchain
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.