influxdata / influxdata/telegraf
outputs.stackdriver: Report dropped metrics with PartialWriteError instead of counting them as written
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Feature Request
Report permanently rejected metrics from `outputs.stackdriver` using
`internal.PartialWriteError`, so the agent accounts for them instead of counting
them as written.
### Proposal
`CreateTimeSeries` partially succeeds: the valid points in a chunk are written
and the invalid ones are rejected in the same call. When the plugin decides a
rejection is permanent it drops those series and returns `nil` from `Write`,
because returning an error would make Telegraf replay a batch that can never
succeed. `nil` means success to the agent, so the entire batch is credited to
`metrics_written`.
Observed with a 250-metric batch where the first chunk of 200 was rejected:
```
E! Dropped 200 time series permanently rejected by Stackdriver
D! Wrote batch of 250 metrics in 9.897625ms
```
250 counted as written, 200 discarded. `metrics_dropped` never moves and the
buffer stays flat, so every self-monitoring signal reports a healthy output
while data is being thrown away. The only evidence is a log line, which makes
this class of loss hard to alert on.
`internal.PartialWriteError` already models exactly this:
```go
type PartialWriteError struct {
Err error
MetricsAccept []int
MetricsReject []int
MetricsRejectErrors []error
}
```
Returning it would let the agent drop the rejected metrics without counting them
as written, increment `metrics_rejected`, and nack tracking metrics correctly
for inputs that use delivery tracking. It would also allow a retryable failure
to replay only the points that were not written, rather than re-sending points
that already landed — which is the duplicate amplification described in #19611.
### Current behavior
`Write` returns `nil` after permanently dropping series, so the agent counts the
whole batch as successfully written.
### Desired behavior
`Write` returns a `PartialWriteError` identifying which metrics were accepted and
which were rejected, with the per-metric reason.
### Use case
Detecting silent data loss from self-monitoring rather than from logs. At high
cardinality a steady trickle of rejected points is invisible today unless someone
greps the Telegraf log.
### Implementation note
The obstacle is index mapping. `timeSeriesBuckets` holds bare
`*monitoringpb.TimeSeries` with no back-reference to the source metric, and the
mapping is many-to-one: a metric fans out to one series per field, plus the
`:counter` duplicate under official naming. The indices have to be threaded
through bucket construction and chunking.
Worth noting that the API response already carries per-series detail, so the
rejected subset is identifiable rather than having to be inferred. A real
rejection looks like:
```
rpc error: code = FailedPrecondition desc = One or more TimeSeries could not be written:
timeSeries[2] (metric.type="...", metric.labels={...}, resource.type="global", ...):
write for resource failed: One or more points were written more frequently than the
maximum sampling period configured for the metric.
```
### Related: call-scoped rejections
Some rejections are independent of the payload and would fail every chunk in a
batch identically, so issuing one request per chunk is wasted work. Detecting
those by enumerating status codes is brittle, and the codes that are genuinely
payload-independent (`UNAUTHENTICATED`, `PERMISSION_DENIED`, `NOT_FOUND`) are
better treated as retryable anyway, which is what #19612 does.
The robust form is behavioural: if consecutive chunks are rejected identically,
stop issuing requests for that batch. That is only safe once the per-series
accounting above exists, because stopping early otherwise forfeits the valid
points in the chunks never attempted. Recording it here so the two land together.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the outputs.stackdriver Write entry point and trace timeSeriesBuckets through bucket construction and chunking. Read internal.PartialWriteError and the API response handling to understand how accepted and rejected series map back to source metrics. Done means permanent rejections return per-metric accounting without counting discarded metrics as written, while identical call-scoped failures stop further chunks safely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- backend, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100