Distributor fails to push data to ingesters during rollout
Open
ops
type/bug
v1
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
It's been noticed that during a rollout, distributor might be unable to push data to ingesters due to the replicas unavailability:
> push.v1.PusherService/Push (500) 2.092304ms Response: "{\"code\":\"unknown\",\"message\":\"at least 2 live replicas required, could only find 1\"}"
This should not be the case, because ingester pods are updated one by one. We need to make sure readiness and health checks are synchronised with the ring status, so that a pod reports about readiness only after it successfully registers in the ring (and the topology has been updated, so all distributors see the ingester).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.