Backpressure: overly susceptible to temporary issues

### Environment

SaaS (https://sentry.io/)

### Steps to Reproduce

Over the last 30 days, we've experienced ~265 instances where backpressure has been marked as unhealthy due to a connection timeout when checking the health of a redis or rabbitmq cluster: https://cloudlogging.app.goo.gl/KNZDAduqrHWQn5At7


Each of these come with a corresponding pause and delay in ingestion:

1 timeout seems to trigger about 15s of ingestion latency.

There can also be instances where multiple trigger in succession, which seems to be enough to trigger a backlog large enough that it may page SRE while it burns down the backlog:


### Expected Result

Some possible improvements we can make:

* Add some retry functionality to avoid flakes
* Require multiple events in a row to trigger the unhealthy state
* Have backpressure fail open instead of closed (could have negative impact if the failures are caused by a real outage of a cluster).

I would probably start with adding retries on failure as it seems like the simplest thing that can work.

### Actual Result

Backpressure pauses ingestion from a single failure.

### Product Area

Ingestion and Filtering

### Link

_No response_

### DSN

_No response_

### Version

_No response_