influxdata / influxdata/telegraf
outputs(wavefront): inline retry logic may log `"Non-retryable error during Wavefront.Write: <nil>"`
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
I'm noticed log lines like `Non-retryable error during Wavefront.Write: ` in Telegraf v1.28.2.
There's a code path in `(*Wavefront).Write` where `err` can be reassigned to `nil` on line 154 if a second call to `w.sender.SendMetric` succeeds. If the 2nd attempt does not return an error, we don't want to log anything about an error.
https://github.com/influxdata/telegraf/blame/f4c56e1597956cd95eb37336e0c08ae361338aba/plugins/outputs/wavefront/wavefront.go#L146
The retry/flush logic has gotten complicated in this method. I'm a little uncertain about the flushing and returning on line 150, and about line 157 returning an error mid-batch.
I think the mid-batch error causes Telegraf to retry the entire batch on the next Write, and I think that may mean successfully sent metrics, unsendable metrics, and unsent metrics all get retried.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in plugins/outputs/wavefront/wavefront.go, especially (*Wavefront).Write around lines 146-157, and trace the calls to w.sender.SendMetric. Check how the second attempt, flushing, and mid-batch returns affect retries. Done means a successful retry does not log a nil error and the resulting batch behavior is verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100