Add exponentional back-off algorithm and addtional error reporting for firehose/stubstreams connections

Do you want to request a feature or report a bug?

bug

What is the current behavior?

When there is an error (eg timeout) on firehose, graph-node waits 2 minutes and then tries again. If the error happens again, it again waits 2 minutes, and so on. If the error is due to software, then this just keeps load on the firehose server forever.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

There was error in firehose code that demonstrates this problem. The problem has since been fixed in firehose, but can reproduced using the old version. Basically can reproduced by simply having firehose never return any data and drop the connection after a while.

What is the expected behavior?

graph-node to have an exponential retry algorithm (eg 30s, 1m, 2m, 4m, 8m, 16m) instead of the hard-coded 2min.

Additionally there the prometheus exporter in graph-node should be improved to report connections retry interval (so alerts can be generated on problem connections)

Begin at graph-node’s firehose/stubstreams connection retry handling and the Prometheus exporter. Reproduce the dropped-connection/no-data case described, then trace the current fixed two-minute wait and existing connection metrics. Done means retries use increasing intervals and the exporter exposes retry interval information suitable for alerts.