Unreachable Providers (server): Provider-Record Prioritization

### Checklist

- [X] My issue is specific & actionable.
- [X] I am not suggesting a protocol enhancement.
- [X] I have searched on the [issue tracker](https://github.com/ipfs/kubo/issues?q=is%3Aissue) for my issue.

### Description

## Context

Our [recent measurements](https://github.com/protocol/network-measurements/issues/49) show that IPFS has trouble fulfilling its baseline practical use case of hosting static websites.

The reasons are at least twofold. On the one hand, large content providers still have trouble announcing all their CIDs to the DHT, and on the other hand, the existing provider records point to unreachable peers. This issue addresses the second reason.

As an example, the below graph showcases the unique providing peers as identified by distinct [PeerIDs](https://docs.libp2p.io/concepts/fundamentals/peers/#peer-id) discovered throughout a specific day in the IPFS DHT for [`ipld.io`](https://ipld.io) ([source](https://probelab.io/websites/ipld.io/#website-trend-providers-ipldio)).


The graph shows that >70% of provider records point to peers that are not reachable. The remaining peers are mainly only reachable via a relaying peer. This either increases latency, if the traffic is relayed or increases time to connect, if the relay is used to facilitate a hole punch.

This trend is not unique to `ipld.io` but a general theme among all our measured websites: https://probelab.io/websites/.

## Problem

When a peer tries to access a website over IPFS, it looks up the provider records in the DHT and tries to connect to the returned peers with a concurrency factor of 10. Due to the large number of provider records that point to peers that are long gone, the peer likely receives such records and therefore will time out in an attempt to establish a connection with them. This significantly lengthens the resolution process to the extent that the whole operation potentially times out (that's a hypothesis).

## Proposal

We identified two ways forward to alleviate the above issue.

1. A prioritization logic of provider records on the server side. The peers that serve provider records sort them in such a way that, e.g., the first one in the list likely contains a peer that is actually reachable.
2. A delayed provider record publication. E.g. only announce blocks if a peer was online for some time. The assumption is that this will filter out rather short-lived peers.

**This GH issue is for proposal 1).**

_The corresponding issue for 2) is #9984._

## Provider Record Prioritization

There was a conversation on 2023-06-19 between ProbeLab and some of the Kubo maintainers where the following strategy was proposed:

We tally consecutive reprovides for each `(PeerID, CID)` tuple. When another peer looks up a certain CID, it gets served the peers with the highest number of consecutive reprovides. We use this number as a proxy for the uptime of a peer. Another factor we could account for is the type of Multiaddresses a peer has announced. We should prioritize un-relayed peers over relayed ones.

There are a few things to consider (non-exhaustive list):

- We could overwhelm stable peers. By always serving the same stable peers as providers for a certain CID, we could put an undesirable amount of load on them. You could also argue that this leads to some kind of centralization.
A potential mitigation could be to define an upper limit a counter can reach. In case of a tie, the servers could return the peers in random order.
- Daily periodicity of peer presence could lead to false positive counter increases. It could happen that peers that exhibit daily periodic uptime, e.g., are booted only in the morning for a few hours and then shut down could end up as stable peers because the provider record TTL is 24h.
- A nice side-effect could be that this approach decreases the risk of unintentional CID eclipsing. If a peer is constantly restarting and generating a new peer ID (but using the same provider store) it would _eclipse_ the CID with bogus records. The proposed logic would deprioritize these provider records. Bad actors could still quite easily generate peer IDs and refresh records at the right times at a low cost.
- How do network topology changes affect the counting of reprovides? If a peer is among the 20 closest to a CID at $t_0$ it is not necessarily among them at $t_0 + 22$ hours (the reprovide interval). This could decrease the effectiveness of this proposed solution.

## Concrete Proposal

I think a concrete proposal fosters efficient discussion. Here's my take:

Count the number of consecutive reprovides. The counter can be `3` at max. Return the provider records in an order that prioritizes a high counter value. In case of a tie, prioritize un-relayed peers. In case there's still a tie, randomize the order with each response. A counter increase is only allowed every ReprovideInterval/2.

## Measurements

TBD: How can we substantiate the proposal with numbers? [some ideas](https://github.com/protocol/network-measurements/issues/49#issuecomment-1599081967)

## References

- https://github.com/protocol/network-measurements/issues/49
- https://probelab.io/websites/
- https://www.notion.so/pl-strflt/Unreachable-Providers-1056416abfef4f1e918c79d55fa473ab?pvs=4