cloudnative-pg / cloudnative-pg/plugin-barman-cloud

Sidecar metrics collection over plugin gRPC has no deadline — one stuck operation hangs the whole instance /metrics

Aperta
#1,045 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug
Lingua principale
Go
Stelle
191
Fork
72
Merge medio
2g 21h
PR unite (30g)
21

Descrizione

## Environment

CloudNativePG 1.30.0, plugin-barman-cloud v0.14.0, PostgreSQL 17/18 clusters, VersityGW (posix) S3 endpoint.

## What happens

The instance sidecar collects plugin metrics over the plugin gRPC with **no deadline on the collect path**. Consequence: if any plugin operation gets stuck (in my case a retention/catalog-maintenance run that never completes against a posix-backend S3 gateway — filed separately), the next scrape blocks on the collector and the **entire** `/metrics` endpoint of that instance hangs indefinitely. Prometheus marks the target down and every metric of that instance disappears, not just the plugin's own collectors.

Observed in production: the primary instance's exporter dark for 6.7 hours while the database itself was healthy — WAL-archiver and backup-staleness alerting was blind precisely on the instance where it matters.

## Expected

- A per-collect deadline (a few seconds) on the plugin metrics gRPC call.
- On timeout/error: return the rest of the metrics plus an error counter (e.g. `..._collector_errors_total`), instead of hanging the whole endpoint. A misbehaving plugin should degrade its own metrics, never the instance's.

## Reproduction sketch

1. ObjectStore against VersityGW (posix backend) with any `retentionPolicy` set.
2. Wait for a retention run to start (~30 min cadence) — it never completes against this backend.
3. Scrape the instance sidecar's metrics port: the request hangs until the client gives up; `up` goes to 0 for the instance.

Companion issue describes the retention hang itself; this one is about the missing deadline that turns any such hang into a full metrics outage.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Traccia il percorso di collect delle metriche gRPC del plugin dell’instance sidecar e riproduci il problema con un’operazione del plugin bloccata, quindi esamina come l’endpoint /metrics gestisce gli errori del collector. Verifica che un timeout per collect consenta all’endpoint di restituire le metriche rimanenti e registrare un errore quando la chiamata al plugin si blocca, mentre le metriche del plugin funzionanti continuano a funzionare.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
go, grpc
Ambito
backend, observability
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
55/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.