influxdata / influxdata/telegraf

outputs(wavefront): inline retry logic may log `"Non-retryable error during Wavefront.Write: <nil>"`

Open
#14,055 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

I'm noticed log lines like `Non-retryable error during Wavefront.Write: ` in Telegraf v1.28.2.

There's a code path in `(*Wavefront).Write` where `err` can be reassigned to `nil` on line 154 if a second call to `w.sender.SendMetric` succeeds. If the 2nd attempt does not return an error, we don't want to log anything about an error.

https://github.com/influxdata/telegraf/blame/f4c56e1597956cd95eb37336e0c08ae361338aba/plugins/outputs/wavefront/wavefront.go#L146

The retry/flush logic has gotten complicated in this method. I'm a little uncertain about the flushing and returning on line 150, and about line 157 returning an error mid-batch.

I think the mid-batch error causes Telegraf to retry the entire batch on the next Write, and I think that may mean successfully sent metrics, unsendable metrics, and unsent metrics all get retried.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in plugins/outputs/wavefront/wavefront.go, especially (*Wavefront).Write around lines 146-157, and trace the calls to w.sender.SendMetric. Check how the second attempt, flushing, and mid-batch returns affect retries. Done means a successful retry does not log a nil error and the resulting batch behavior is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.