influxdata / influxdata/telegraf

prometheusremotewrite doesn't send stale markers

Open
#16,641 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
[agent]
interval = "10s"
flush_interval = "10s"

[[inputs.mock]]
metric_name = "mock"
[[inputs.mock.step]]
name = "plus_one"
step = 1.0

[[processors.starlark]]
source = '''
def apply(metric):
metric.tags["minute"] = str(metric.time)[:8]
return metric
'''

[[outputs.http]]
url = "http://prometheus:9090/api/v1/write"
method = "POST"
data_format = "prometheusremotewrite"
[outputs.http.headers]
Content-Type = "application/x-protobuf"
Content-Encoding = "snappy"
X-Prometheus-Remote-Write-Version = "0.1.0"
```

### Logs from Telegraf

```text
n/a
```

### System info

Telegraf 1.34

### Docker

_No response_

### Steps to reproduce

1. Using this docker-compose package, run `docker compose up`
2. Visit http://localhost:3000 to see the dashboard after a few minutes

[staleness-demo.zip](https://github.com/user-attachments/files/19236039/staleness-demo.zip)

### Expected behavior

According to the Prometheus remote write spec:

> Prometheus remote write compatible senders MUST send stale markers when a time series will no longer be appended to.

### Actual behavior

This was mentioned years ago in #9166, where @ssoroka said it's not possible to fix. But I want to challenge that assertion, and provide an example of why this is problematic.

With this example docker-compose package, Telegraf generates a mock series that increments every 10 seconds. It applies a label with the current timestamp in _minutes_ which means a new series is born every minute. In Grafana you can see each series flatlines after a minute, and then remains for 5 minutes before disappearing.

![Image](https://github.com/user-attachments/assets/b7e8c70c-18b8-4f4f-8c4f-21d2f256af69)

So at any moment in time, you could have 3 or 4 overlapping series, when there should be only 1. This causes unpredictable and confusing behavior in Grafana, especially with state timelines for info metrics.

I propose to handle it like this:
1. Telegraf knows the interval for each input, in this example 10s.
2. If that amount of time passes without receiving a sample, Telegraf should write a stale marker for that series to the Prometheus output.

### Additional info

_No response_

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by tracing the prometheusremotewrite output and reviewing the Prometheus remote write staleness requirement alongside issue #9166. Use the supplied docker-compose reproduction to determine how series lifetimes are tracked, and consider the work done when inactive series receive correct stale markers without disrupting active series.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.