influxdata / influxdata/telegraf

[outputs.influxdb_v2] Apply retry wait when connection refused

Open
#17,122 1 comment 1 reaction 0 assignees View on GitHub
feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

Running `telegraf` with multiple `urls` for `outputs.influxdb_v2` for high availability.

### Expected behavior

When one of the endpoints is unreachable (`syscall.ECONNREFUSED`), it should be handled with the same `retryTime` logic that applies to explicit http states such as `http.StatusServiceUnavailable` to avoid futile connection attempts waiting for timeouts.

### Actual behavior

When one of the endpoints is down (`connect: connection refused`), `outputs.influxdb_v2` continues to try and write to the unreachable `url` in the randomized loop for writing to multiple `urls`.
As a result, it will frequently wait for the timeout, leading to a delay in metrics transmission.

### Additional info

I believe this can simply be added to the existing logic:
```
diff --git a/plugins/outputs/influxdb_v2/http.go b/plugins/outputs/influxdb_v2/http.go
index 710656c26..43d58b504 100644
--- a/plugins/outputs/influxdb_v2/http.go
+++ b/plugins/outputs/influxdb_v2/http.go
@@ -19,6 +19,7 @@ import (
"strings"
"sync"
"sync/atomic"
+ "syscall"
"time"

"github.com/alitto/pond/v2"
@@ -328,6 +329,14 @@ func (c *httpClient) writeBatch(ctx context.Context, b *batch) error {
resp, err := c.client.Do(req.WithContext(ctx))
if err != nil {
internal.OnClientError(c.client, err)
+ if errors.Is(err, syscall.ECONNREFUSED) {
+ retryDuration := getRetryDuration(resp.Header, c.retryCount.Add(1))
+ return &ThrottleError{
+ Err: fmt.Errorf("failed to write to %s; will retry in %s. (%s)", b.bucket, retryDuration, resp.Status),
+ StatusCode: resp.StatusCode,
+ RetryAfter: retryDuration,
+ }
+ }
return err
}
defer resp.Body.Close()

```

Contributor guide

Open the contributing guide

Research direction

Start in plugins/outputs/influxdb_v2/http.go at httpClient.writeBatch and compare connection-refused handling with the existing retry logic for HTTP status responses. Confirm that an ECONNREFUSED failure uses retryTime before another endpoint attempt, and verify the outputs.influxdb_v2 behavior with the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, networking
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.