prometheus / prometheus/client_python

Add an option to expose the metric timestamps from the Prometheus format

Aperta
#969 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Lingua principale
Python
Stelle
4.4k
Fork
876
Merge medio
8g 4h
PR unite (30g)
1

Descrizione

Spin off from https://github.com/prometheus/client_python/pull/967 and https://github.com/prometheus/client_python/issues/847

Background

The Prometheus exposition format (https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#comments-help-text-and-type-information) allows exporters to specify an optional timestamp for each sample. If this is unset, the collector uses the timestamp that the sample is collected.

This timestamp is useful in a certain situation. My concrete use case is to expose GitHub API rate limit as a metric. This API rate limit is a value that decreases over time, and periodically it resets to the maximum value. We can get this API rate limit value as a HTTP response header when calling GitHub API.

Imagine a situation where there are two (physical) servers making this GitHub API call, and for the sake of illustration, let's assume that one server issues more requests and the other one does less.

  Server1 (frequently issue requests) Server2 (somehow less requests)
Observed API limit 100 4900
(Last time a server made a GH API call) 3 minutes ago 50 minutes ago

From human's point of view, because server1 made an API call more recently than server2, we can tell that server1's observed API limit is the current value. However, without exposing the sample timestamp, Prometheus cannot distinguish which one is the most recent value. This is because when Prometheus collects sample from these two servers, when there's no timestamp specified, it treats these samples as "the samples collected now".

By exposing the sample timestamp, Prometheus should be able to treat these two samples correctly, and it can tell that server1' s value is the most recent value.

Request

With https://github.com/prometheus/client_python/pull/967, client_python should learn timestamps for Gauge metrics in the multiprocessing mode. This issue is asking for an option to expose those timestamps via Prometheus format.

Code pointers

Sample already can take a timestamp (https://github.com/prometheus/client_python/blob/249490e4ed30d4266182cfce50fe7046a484affd/prometheus_client/samples.py#L49), and so as the Prometheus exposition formatter (https://github.com/prometheus/client_python/blob/249490e4ed30d4266182cfce50fe7046a484affd/prometheus_client/exposition.py#L191-L194).

For the multiprocessing mode, changing the metrics aggregation logic to propagate the timestamps (https://github.com/prometheus/client_python/blob/249490e4ed30d4266182cfce50fe7046a484affd/prometheus_client/multiprocess.py#L146) would suffice (based on a flag or something).

For the non-multiprocessing mode, need to change this _child_samples (https://github.com/prometheus/client_python/blob/249490e4ed30d4266182cfce50fe7046a484affd/prometheus_client/metrics.py#L433-L434) to take a timestamp.

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia da prometheus_client/samples.py ed exposition.py per tracciare come vengono rappresentati e formattati i timestamp dei sample. Poi esamina metrics.py::_child_samples e multiprocess.py relativamente alla logica di aggregazione citata, includendo la pull request e l’issue correlate. Il lavoro è completato quando un’opzione espone i timestamp nell’output di Prometheus e li preserva sia in modalità multiprocessing sia in modalità non-multiprocessing.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
prometheus, python
Ambito
observability
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
42/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.