Beacon node hedge

🎯 Problem to be solved

Currently we have two ways of specifying extra beacon nodes:

1. Add multiple beacon nodes to the list in the beacon-node-endpoints flag. We call all of the beacon nodes simultaneously and use the response of the first one.
2. Add beacon nodes as fallback beacon nodes in the fallback-beacon-node-endpoints flag. If all of the beacon nodes specified in the beacon-node-endpoints flag fail, we call all of the fallback beacon nodes simultaneously and use the response of the first one.

If operators want to have some fault tolerance on the BN they use, they can proceed with the first option.
If operators want to have some backup and use external BN in rare occasions where all of their BNs and keep their validator running while they fix some issue, they can proceed with the second option.
A mix of the two works great as well.

In huge clusters, more often than not the following prerequisites are apparent:

• if a beacon node fails to respond to a request, it's usually when it is under too much load;
• having multiple beacon nodes under beacon-node-endpoints is of not much help as it will only put multiple beacon nodes under the same big load;
• fallback beacon nodes are triggered only after a timeout, which is by default 2 seconds; this can be way too late for a lot of calls, especially those that take longer.

That said, there is no great way to hedge against slow beacon node with the current setup.

🛠️ Proposed solution

• Approved design doc: link
• Core team consensus on the proposed solution

Add a hedge-fallback-beacon-nodes. If enabled, fallback beacon nodes will be called if the regular beacon nodes do not supply the required data in ideally expected time, which should be shorter than the regular timeout (i.e.: timeout/4 or 0.5s by default).

Note that the request from the regular beacon nodes should not be abandoned, meaning that if Charon calls all the regular beacon nodes at T, the hedged fallback beacon nodes at T+0.5s, Charon should wait for response from either. If the regular beacon nodes supply the response in T+0.6s and fallbacks at T+0.7s, Charon should not wait until T+0.7s.

Optionally, consider if the fallback beacon nodes should be called sequentially with + hedge for each call and not simultaneously as they are currently. Meaning all regular beacon nodes are called at T, first fallback beacon node is called at T+0.5, second fallback beacon node at T+1.0, third fallback beacon node at T+1.5, etc.

🧪 Tests

• Tested by new automated unit/integration/smoke tests
• Manually tested on core team/canary/test clusters
• Manually tested on local compose simnet

No implementation files, tests, or entry points are identified. Start by resolving the linked design document and obtaining core-team consensus on hedge timing, fallback behavior, and cancellation semantics; done means the approved behavior is implemented and covered by automated and manual tests.