grafana / grafana/pyroscope

Distributor fails to push data to ingesters during rollout

Open
#2,702 0 comments 2 reactions 0 assignees View on GitHub
ops type/bug v1
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

It's been noticed that during a rollout, distributor might be unable to push data to ingesters due to the replicas unavailability:

> push.v1.PusherService/Push (500) 2.092304ms Response: "{\"code\":\"unknown\",\"message\":\"at least 2 live replicas required, could only find 1\"}"

This should not be the case, because ingester pods are updated one by one. We need to make sure readiness and health checks are synchronised with the ring status, so that a pod reports about readiness only after it successfully registers in the ring (and the topology has been updated, so all distributors see the ingester).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.