Issue with DNS Caching After Internet Outage in go-livepeer Node

Nobody has claimed this yet.

Issue Description

Hello,

I've encountered an issue with my orchestrator node using go-livepeer. After a scheduled maintenance by my internet provider, my orchestrator node didn't resume its service normally and got stuck in a loop with the following error:

E1203 11:40:11.668550 1 block_watcher.go:123] blockwatch.Watcher error encountered - trying again on next polling interval err="Post \"https://arb-mainnet.g.alchemy.com/v2/YOUR_API_KEY\": context deadline exceeded"

To restore the connection to the RPC, I had to manually restart the Docker container (version 0.8.0). This problem occurs with different RPC providers, including Alchemy.
Interestingly, one of my nodes (AI) that I forgot to restart eventually managed to reconnect to the RPC after approximately 15 minutes.

Suspected Cause

Mike suggested that the issue might be related to DNS caching. When the internet connection is restored, the DNS entries may be outdated, causing the node to fail to reconnect properly until the container is restarted.

Proposed Solution

One proposed solution is to adjust the health check endpoint to test the node's ability to stream and block poll RPC requests. If the health check detects issues, the Docker container could be configured to automatically restart, ensuring the node attempts to reconnect without manual intervention.

Thank you!

Start by reading block_watcher.go at line 123 and the Docker health-check configuration, then trace the RPC polling and streaming behavior after connectivity returns. Done means the health check detects failed RPC connectivity and the container recovers without a manual restart.

Written by the indexing model from the issue text.