Automattic / Automattic/pingo

Metrics should report timestamps

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
39
Forks
4
PR merge metrics
No merged PRs in 30d

Description

In non-triggered dump mode, the metrics endpoint returns data "from the past" without any indication that this is the case, let alone which exact point in time it's from.

Take as an example a Pingo instance with multiple flows configured. Each flow gets its own dumper, which run independently from each other. Each dumper will sleep for 15 seconds, then do a bit of work, and sleep again for 15 seconds; which means one "dump cycle" is 15s sleep plus however long processing takes. Depending on what rate the flow is configured for, the amount of data gathered in that 15s sleep interval varies, and with it the time for processing. Because of this over time the cycles of the dumpers can (and will) shift. This by itself is fine and not a problem per-se.

Scrapers usually run on a interval, e.g. the trigger fires, it does some work, and then sleeps 15s minus the work-time until the next trigger; so a scrape cycle is very much exactly 15s. Since the dump cycle is longer and also varies, the scrape cycle will also constantly shift with regards to the dump cycles. That is per-se also not a problem, you may simply rarely read some data twice.

It however becomes a problem when the metrics don't contain timestamps. Take the example below:

Image

The scraper runs at t37, shortly after Dumper 2 ran but just before Dumper 1 could/did. It will read data Dumper 2 created at t36 describing the time-window t21 to t36, and Dumper 1 created at t22 describing the time-window t7 to t22; so "vastly" different time windows. Because the metrics data does not contain any timestamps, the scraper has to no chance to know what 15s time-window those numbers actually describe; it could be any of the possible 15s time-windows.

You run into a similar problem if the scraper runs at a higher frequency, for example 5s interval. For some flows it may read new data because in the meantime the respective dumper woke up, but for others it will read the same data again. Since there are no timestamps it has no way to detect which is which. If the value is different it must be new, if the value is the same it could be old or new but unchanged.

The [Prometheus Exposition Text Format](https://prometheus.io/docs/instrumenting/exposition_formats/#prometheus-text-format) allows to optionally specify a timestamp for each sample (data point). So all of this can be solved by having each dumper track the last time they ran (which they already do anyway), and have the metrics endpoint attach that timestamp to any samples from that dumper. That way you know exactly which point in time the metric was generated and what time-window it describes.

Triggered dump is not effected, since there the metric data will be generated fresh for every scrape, covering the time-window since the last scrape. Timestamps are not necessary here, since they will be (very close to) "now" anyway.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the metrics endpoint and the per-flow dumper logic described in the issue, then review Prometheus's exposition format for sample timestamps. Done means non-triggered metrics include the timestamp of the dumper's last run, while triggered-dump metrics remain unchanged; verify behavior across independently running dumpers and repeated scrapes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.