Broker resiliency to misbehaving historical nodes

Sometimes we see 'zombie' nodes that are nominally responsive but are having underlying problems. This can be due to bad disks, bad configuration, or any number of other causes. Due to the vicissitudes of life, we cannot necessarily predict all of these in advance. So two things would be useful as general mitigations,

1. An ability for the broker to retry queries to data nodes that fail, on the grounds that perhaps another node will succeed.
2. An ability for the broker to blacklist data nodes that fail too often relative to other nodes.

You want (1) to not be too aggressive -- it could lead to doing too much work on a query that is doomed to failure anyway (maybe something's wrong with the query). You also want (2) to not be too aggressive -- it's senseless to blacklist half the cluster, for example.

You also want the list from (2) to be exposed via API somehow, since folks might want to build automation that takes those nodes out of service, raises alerts about them, replaces them automatically, etc.

No files, tests, or entry points are named. Start by locating the broker path that queries historical data nodes, then determine how retries and repeated failures should be bounded. Done means defining both behaviors and exposing the blacklist through an API suitable for monitoring or automation.

Written by the indexing model from the issue text.