hyperledger-firefly / hyperledger-firefly/ethconnect

Increase checkpoint block number when there are no events, to minimize cost on restart

Open
#105 0 comments 0 reactions 1 assignee Claimed by @peterbroadhurst View on GitHub
Dominant language
Go
Stars
76
Forks
31
PR merge metrics
No merged PRs in 30d

Description

Currently we only increase the block high-watermark on a subscription to an event stream, when we have successfully processed a batch:
https://github.com/kaleido-io/ethconnect/blob/ad40de61370ec2b99a8cfc79cdf72e5dcac3d914/internal/kldevents/logprocessor.go#L83

This means that if no events arrive on a subscription for a long time (millions of blocks) we never increase the blockHWM, and write a new checkpoint. So on restart (of ethconnect or the node), we drive a high load on the node as we recreate the filter at a very old block number.

Instead we need a system to safely move the blockHWM forwards, at regular intervals, so it's never more than a few hundred blocks behind the head. This will mean periodically in the polling cycle, asking the node for the current block height ahead of the call to `eth_getFilterChanges` and if no results are found, then setting the blockHWM to a safe number (such as 100 blocks behind the head, if that's higher than zero, and higher than the previous blockHWM).

fyi @vdamle @panghalamit

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.