Add exponentional back-off algorithm and addtional error reporting for firehose/stubstreams connections

Do you want to request a feature or report a bug?

bug

What is the current behavior?

When there is an error (eg timeout) on firehose, graph-node waits 2 minutes and then tries again. If the error happens again, it again waits 2 minutes, and so on. If the error is due to software, then this just keeps load on the firehose server forever.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

There was error in firehose code that demonstrates this problem. The problem has since been fixed in firehose, but can reproduced using the old version. Basically can reproduced by simply having firehose never return any data and drop the connection after a while.

What is the expected behavior?

graph-node to have an exponential retry algorithm (eg 30s, 1m, 2m, 4m, 8m, 16m) instead of the hard-coded 2min.

Additionally there the prometheus exporter in graph-node should be improved to report connections retry interval (so alerts can be generated on problem connections)

Inizia dalla gestione dei tentativi di riconnessione di graph-node a firehose/stubstreams e dal Prometheus exporter. Riproduci il caso descritto di connessione interrotta senza dati, quindi traccia l’attesa fissa attuale di due minuti e le metriche di connessione esistenti. Il lavoro è completato quando i tentativi utilizzano intervalli crescenti e l’exporter espone informazioni sull’intervallo tra i tentativi adatte agli alert.