influxdata / influxdata/telegraf
prometheusremotewrite doesn't send stale markers
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Relevant telegraf.conf
```toml
[agent]
interval = "10s"
flush_interval = "10s"
[[inputs.mock]]
metric_name = "mock"
[[inputs.mock.step]]
name = "plus_one"
step = 1.0
[[processors.starlark]]
source = '''
def apply(metric):
metric.tags["minute"] = str(metric.time)[:8]
return metric
'''
[[outputs.http]]
url = "http://prometheus:9090/api/v1/write"
method = "POST"
data_format = "prometheusremotewrite"
[outputs.http.headers]
Content-Type = "application/x-protobuf"
Content-Encoding = "snappy"
X-Prometheus-Remote-Write-Version = "0.1.0"
```
### Logs from Telegraf
```text
n/a
```
### System info
Telegraf 1.34
### Docker
_No response_
### Steps to reproduce
1. Using this docker-compose package, run `docker compose up`
2. Visit http://localhost:3000 to see the dashboard after a few minutes
[staleness-demo.zip](https://github.com/user-attachments/files/19236039/staleness-demo.zip)
### Expected behavior
According to the Prometheus remote write spec:
> Prometheus remote write compatible senders MUST send stale markers when a time series will no longer be appended to.
### Actual behavior
This was mentioned years ago in #9166, where @ssoroka said it's not possible to fix. But I want to challenge that assertion, and provide an example of why this is problematic.
With this example docker-compose package, Telegraf generates a mock series that increments every 10 seconds. It applies a label with the current timestamp in _minutes_ which means a new series is born every minute. In Grafana you can see each series flatlines after a minute, and then remains for 5 minutes before disappearing.

So at any moment in time, you could have 3 or 4 overlapping series, when there should be only 1. This causes unpredictable and confusing behavior in Grafana, especially with state timelines for info metrics.
I propose to handle it like this:
1. Telegraf knows the interval for each input, in this example 10s.
2. If that amount of time passes without receiving a sample, Telegraf should write a stale marker for that series to the Prometheus output.
### Additional info
_No response_
Contributor guide
Research direction
No source files or tests are named. Start by tracing the prometheusremotewrite output and reviewing the Prometheus remote write staleness requirement alongside issue #9166. Use the supplied docker-compose reproduction to determine how series lifetimes are tracked, and consider the work done when inactive series receive correct stale markers without disrupting active series.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100