Add exponentional back-off algorithm and addtional error reporting for firehose/stubstreams connections

Do you want to request a feature or report a bug?

bug

What is the current behavior?

When there is an error (eg timeout) on firehose, graph-node waits 2 minutes and then tries again. If the error happens again, it again waits 2 minutes, and so on. If the error is due to software, then this just keeps load on the firehose server forever.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

There was error in firehose code that demonstrates this problem. The problem has since been fixed in firehose, but can reproduced using the old version. Basically can reproduced by simply having firehose never return any data and drop the connection after a while.

What is the expected behavior?

graph-node to have an exponential retry algorithm (eg 30s, 1m, 2m, 4m, 8m, 16m) instead of the hard-coded 2min.

Additionally there the prometheus exporter in graph-node should be improved to report connections retry interval (so alerts can be generated on problem connections)

Beginne bei der Wiederholungsbehandlung für Verbindungen von graph-node zu firehose/stubstreams und beim Prometheus exporter. Reproduziere den beschriebenen Fall einer abgebrochenen Verbindung ohne Daten und verfolge anschließend die aktuelle feste Wartezeit von zwei Minuten sowie die vorhandenen Verbindungsmetriken. Als abgeschlossen gilt die Aufgabe, wenn Wiederholungsversuche steigende Intervalle verwenden und der exporter Informationen zum Wiederholungsintervall bereitstellt, die für Alerts geeignet sind.