Handle replica corruption in `recoverLocallyUpToGlobalCheckpoint` more gracefully

Today if we discover a shard or translog corruption in `recoverLocallyUpToGlobalCheckpoint` when recovering a replica, we fail the recovery and retry, but the retry is typically on the same node that just failed so it will fail again. Instead we should handle this situation more gracefully by ignoring the local corrupt data and performing a full peer recovery.

---

## Workaround

Set `number_of_replicas: 0` on the affected index, which should clear up the corrupt data. Then set it back to the desired number of replicas.