[Bug] Firehose won't auto remove cursor until it's turned off

まず config.tmpl の firehose エンドポイント設定を確認し、レポートに記載された再起動シーケンスを再現します。その際、EthereumFirehoseBlockIngestor と BlockStore のログをエントリーポイントとして使用します。firehose を有効にした場合と無効にした場合の挙動を比較し、復旧のためにエンドポイントを手動でコメントアウトして graph-node を再起動する必要がなくなっていることを確認します。

Bug report

A scenario where firehose endpoint starts lagging. After recovered then firehose stopped ingesting blocks even after graph-node restarts.

Manual fix :

1. Comment out firehose endpoint from config.tmpl file and restart graph-node by running RPC only.
2. After up and running, uncomment firehose config.tmpl (Turn on firehose)
3. Blocks starts ingesting again with firehose.

Hypothesis :

-Firehose won't auto remove cursor when it's turned on
-After turning off firehose, it shows the delete cursor log

Relevant log output

Logs when firehose is kept on even after restarts (generates every 30 seconds)

Feb 24 17:00:17.777 INFO Trying to reconnect the Blockstream after disconnect, cursor: , endpoint uri: firehose, network_name: bsc, provider: firehose, component: EthereumFirehoseBlockIngestor
Feb 24 17:00:17.787 INFO Starting GraphQL WebSocket server at: ws://localhost:8001, component: SubscriptionServer
Feb 24 17:00:17.873 INFO Blockstream connected, consuming blocks, network_name: bsc, provider: firehose, component: EthereumFirehoseBlockIngestor
-----------------------------------------------------------------------------------------
Logs when firehose is turned off

Feb 24 17:23:26.816 INFO Removed cursor for non-firehose chain, now cleaning shallow blocks, lower_bound: 46928163, network: bsc, component: BlockStore
-----------------------------------------------------------------------------------------
Logs when firehose is turned off then back on

Feb 24 17:27:51.787 INFO Trying to reconnect the Blockstream after disconnect, cursor: , endpoint uri: firehose, network_name: bsc, provider: firehose, component: EthereumFirehoseBlockIngestor

Feb 24 17:27:51.812 INFO Blockstream connected, consuming blocks, network_name: bsc, provider: firehose, component: EthereumFirehoseBlockIngestor

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

• Tick this box if this bug is caused by a regression found in the latest release.
• Tick this box if this bug is specific to the hosted service.
• I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux