Clients fail to send read requests to all replicas

We recently encountered a situation where a two-storage-server team was hot with reads. At some point, one of these storage servers started experiencing noticeably more reads than the other. We decided to exclude the worse one, and after data movement completed we ended up in a state where only a single storage server was hot. A subsequent exclude of the new process had the same outcome.

Eventually we bounced the clients, and load started to distribute evenly between the two storage servers again.

Start by examining the reported client and storage-server read distribution after excluding a server, then compare behavior before and after bouncing clients. Reproduce the two-storage-server scenario if possible and determine what condition prevents reads from reaching both replicas; done means the cause is identified and the issue includes a verified fix or a documented reproduction.

Written by the indexing model from the issue text.