Add exponentional back-off algorithm and addtional error reporting for firehose/stubstreams connections

Do you want to request a feature or report a bug?

bug

What is the current behavior?

When there is an error (eg timeout) on firehose, graph-node waits 2 minutes and then tries again. If the error happens again, it again waits 2 minutes, and so on. If the error is due to software, then this just keeps load on the firehose server forever.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

There was error in firehose code that demonstrates this problem. The problem has since been fixed in firehose, but can reproduced using the old version. Basically can reproduced by simply having firehose never return any data and drop the connection after a while.

What is the expected behavior?

graph-node to have an exponential retry algorithm (eg 30s, 1m, 2m, 4m, 8m, 16m) instead of the hard-coded 2min.

Additionally there the prometheus exporter in graph-node should be improved to report connections retry interval (so alerts can be generated on problem connections)

graph-node の firehose/stubstreams 接続のリトライ処理と Prometheus exporter から着手してください。説明されている接続切断後にデータがないケースを再現し、現在の固定された2分間の待機時間と既存の接続メトリクスを追跡してください。リトライで間隔が段階的に長くなり、exporter がアラートに適したリトライ間隔の情報を公開できれば完了です。