influxdata / influxdata/telegraf

outputs.stackdriver: sendBatch mutates the caller's metrics, so replayed batches lose tags and histogram fields

Open
#19,613 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Relevant telegraf.conf

```toml
[[outputs.stackdriver]]
project = "projects/"
namespace = "test"
tags_as_resource_label = ["job_name"]
```

### Logs from Telegraf

```text
E! [outputs.stackdriver] Unable to build distribution from metric : no sum field present
```

### System info

Telegraf master (d110f806), outputs.stackdriver

### Docker

No response

### Steps to reproduce

1. Configure `outputs.stackdriver` with `tags_as_resource_label` set, or send histogram metrics.
2. Cause a write to fail with a retryable error so Telegraf replays the batch
(e.g. make the endpoint briefly unavailable).
3. Compare the retried request against the first attempt.

### Expected behavior

Replaying a batch should send the same payload as the first attempt.

### Actual behavior

`sendBatch` mutates the caller's metrics in place, and Telegraf's buffer hands
back the same `telegraf.Metric` pointers on replay. So the second attempt sees
metrics that the first attempt already stripped:

- `m.RemoveTag(tag)` for every entry in `TagsAsResourceLabels`
(`plugins/outputs/stackdriver/stackdriver.go:263`) — on retry those tags are
gone, so they are silently missing from the resource labels of the replayed
request.
- `buildHistogram` calls `m.RemoveField("sum")` and `m.RemoveField("count")`
(`:589`, `:599`) — on retry the fields are gone, so the metric fails with
`no sum field present` (`:583`) / `no count field present` (`:593`) and is
dropped entirely.

This is independent of which error triggered the replay, and predates the
classification work in #19611.

### Reproduction

Config, with a transient failure forced on the first write attempt so Telegraf
replays the batch:

```toml
[[outputs.stackdriver]]
project = "projects/shardtest"
namespace = "s"
resource_type = "prometheus_target"
tags_as_resource_label = ["instance", "cluster"]
[outputs.stackdriver.resource_labels]
location = "us-east4-b"
namespace = "store"
job = "Telegraf"
```

Input: `cpu_load,instance=st0429,cluster=st0429,host=probe value=42i `

What the API received on each attempt:

```
ATTEMPT 1 | resource.labels{cluster=st0429, instance=st0429, job=Telegraf, location=us-east4-b, namespace=store, project_id=...}
ATTEMPT 1 -> replying Unavailable (forces Telegraf to retry)

ATTEMPT 2 | resource.labels{job=Telegraf, location=us-east4-b, namespace=store, project_id=...}
ATTEMPT 2 -> OK
```

`instance` and `cluster` are absent from the retry, and the write then succeeds
against the wrong resource. Nothing in the logs indicates this happened.

### Impact

`monitored_resource` types have a required label set, and Cloud Monitoring
rejects a write whose set is incomplete. Confirmed against a live project with
`resource_type = "prometheus_target"` and no resource labels beyond the
`project_id` the plugin sets itself:

```
code = InvalidArgument desc = One or more TimeSeries could not be written:
timeSeries[0] (... resource.type="prometheus_target",
resource.labels={"project_id": "..."}):
the set of resource labels is incomplete, missing (cluster, instance, job, location, namespace)
```

So `prometheus_target` requires six labels, five of which the user supplies. The
same write succeeds once they are present.

That gives the mutation two distinct failure modes on replay, depending on
whether a default exists in `resource_labels` for the tag being promoted:

**No default for that label.** The retry omits it, the resource label set is
incomplete, and the write is rejected with `INVALID_ARGUMENT`. Because that is a
permanent rejection, the metrics are then dropped. A single transient failure is
therefore converted into permanent data loss for everything in that batch: the
first attempt failed for a retryable reason, and the retry can no longer be
formed correctly.

**A default is configured for that label**, which is the natural way to use this
option — a static fallback in `resource_labels`, overridden per metric by
`tags_as_resource_label`. The retry finds the tag gone, leaves the default in
place, and the write is accepted against the wrong resource. Cloud Monitoring
shards its storage backend by resource-label hash, so every retried series
collapses onto the single default resource: ingestion concentrates on one shard,
and those points land under a different resource identity than the rest of their
own series. Nothing in the logs indicates it happened.

For histograms the failure is louder but the cause is the same: `buildHistogram`
removes `sum` and `count`, so the replayed metric is rejected outright with
`no sum field present` and dropped.

A fix would be for `sendBatch` to derive its resource labels and histogram values
without mutating the metric it was handed, so a replay sees the same input the
first attempt did.

Contributor guide

Open the contributing guide

Research direction

Read plugins/outputs/stackdriver/stackdriver.go around sendBatch (line 263) and buildHistogram (lines 583-599), then trace how Telegraf replays the same Metric pointers. Verify that resource-label and histogram handling leave the caller's metrics unchanged, so the retry receives the same tags and fields and produces the same payload as the first attempt.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.