influxdata / influxdata/telegraf
outputs.stackdriver: sendBatch mutates the caller's metrics, so replayed batches lose tags and histogram fields
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Relevant telegraf.conf
```toml
[[outputs.stackdriver]]
project = "projects/"
namespace = "test"
tags_as_resource_label = ["job_name"]
```
### Logs from Telegraf
```text
E! [outputs.stackdriver] Unable to build distribution from metric : no sum field present
```
### System info
Telegraf master (d110f806), outputs.stackdriver
### Docker
No response
### Steps to reproduce
1. Configure `outputs.stackdriver` with `tags_as_resource_label` set, or send histogram metrics.
2. Cause a write to fail with a retryable error so Telegraf replays the batch
(e.g. make the endpoint briefly unavailable).
3. Compare the retried request against the first attempt.
### Expected behavior
Replaying a batch should send the same payload as the first attempt.
### Actual behavior
`sendBatch` mutates the caller's metrics in place, and Telegraf's buffer hands
back the same `telegraf.Metric` pointers on replay. So the second attempt sees
metrics that the first attempt already stripped:
- `m.RemoveTag(tag)` for every entry in `TagsAsResourceLabels`
(`plugins/outputs/stackdriver/stackdriver.go:263`) — on retry those tags are
gone, so they are silently missing from the resource labels of the replayed
request.
- `buildHistogram` calls `m.RemoveField("sum")` and `m.RemoveField("count")`
(`:589`, `:599`) — on retry the fields are gone, so the metric fails with
`no sum field present` (`:583`) / `no count field present` (`:593`) and is
dropped entirely.
This is independent of which error triggered the replay, and predates the
classification work in #19611.
### Reproduction
Config, with a transient failure forced on the first write attempt so Telegraf
replays the batch:
```toml
[[outputs.stackdriver]]
project = "projects/shardtest"
namespace = "s"
resource_type = "prometheus_target"
tags_as_resource_label = ["instance", "cluster"]
[outputs.stackdriver.resource_labels]
location = "us-east4-b"
namespace = "store"
job = "Telegraf"
```
Input: `cpu_load,instance=st0429,cluster=st0429,host=probe value=42i `
What the API received on each attempt:
```
ATTEMPT 1 | resource.labels{cluster=st0429, instance=st0429, job=Telegraf, location=us-east4-b, namespace=store, project_id=...}
ATTEMPT 1 -> replying Unavailable (forces Telegraf to retry)
ATTEMPT 2 | resource.labels{job=Telegraf, location=us-east4-b, namespace=store, project_id=...}
ATTEMPT 2 -> OK
```
`instance` and `cluster` are absent from the retry, and the write then succeeds
against the wrong resource. Nothing in the logs indicates this happened.
### Impact
`monitored_resource` types have a required label set, and Cloud Monitoring
rejects a write whose set is incomplete. Confirmed against a live project with
`resource_type = "prometheus_target"` and no resource labels beyond the
`project_id` the plugin sets itself:
```
code = InvalidArgument desc = One or more TimeSeries could not be written:
timeSeries[0] (... resource.type="prometheus_target",
resource.labels={"project_id": "..."}):
the set of resource labels is incomplete, missing (cluster, instance, job, location, namespace)
```
So `prometheus_target` requires six labels, five of which the user supplies. The
same write succeeds once they are present.
That gives the mutation two distinct failure modes on replay, depending on
whether a default exists in `resource_labels` for the tag being promoted:
**No default for that label.** The retry omits it, the resource label set is
incomplete, and the write is rejected with `INVALID_ARGUMENT`. Because that is a
permanent rejection, the metrics are then dropped. A single transient failure is
therefore converted into permanent data loss for everything in that batch: the
first attempt failed for a retryable reason, and the retry can no longer be
formed correctly.
**A default is configured for that label**, which is the natural way to use this
option — a static fallback in `resource_labels`, overridden per metric by
`tags_as_resource_label`. The retry finds the tag gone, leaves the default in
place, and the write is accepted against the wrong resource. Cloud Monitoring
shards its storage backend by resource-label hash, so every retried series
collapses onto the single default resource: ingestion concentrates on one shard,
and those points land under a different resource identity than the rest of their
own series. Nothing in the logs indicates it happened.
For histograms the failure is louder but the cause is the same: `buildHistogram`
removes `sum` and `count`, so the replayed metric is rejected outright with
`no sum field present` and dropped.
A fix would be for `sendBatch` to derive its resource labels and histogram values
without mutating the metric it was handed, so a replay sees the same input the
first attempt did.
Contributor guide
Research direction
Read plugins/outputs/stackdriver/stackdriver.go around sendBatch (line 263) and buildHistogram (lines 583-599), then trace how Telegraf replays the same Metric pointers. Verify that resource-label and histogram handling leave the caller's metrics unchanged, so the retry receives the same tags and fields and produces the same payload as the first attempt.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100