influxdata / influxdata/telegraf
[outputs.influxdb_v2] Apply retry wait when connection refused
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Use Case
Running `telegraf` with multiple `urls` for `outputs.influxdb_v2` for high availability.
### Expected behavior
When one of the endpoints is unreachable (`syscall.ECONNREFUSED`), it should be handled with the same `retryTime` logic that applies to explicit http states such as `http.StatusServiceUnavailable` to avoid futile connection attempts waiting for timeouts.
### Actual behavior
When one of the endpoints is down (`connect: connection refused`), `outputs.influxdb_v2` continues to try and write to the unreachable `url` in the randomized loop for writing to multiple `urls`.
As a result, it will frequently wait for the timeout, leading to a delay in metrics transmission.
### Additional info
I believe this can simply be added to the existing logic:
```
diff --git a/plugins/outputs/influxdb_v2/http.go b/plugins/outputs/influxdb_v2/http.go
index 710656c26..43d58b504 100644
--- a/plugins/outputs/influxdb_v2/http.go
+++ b/plugins/outputs/influxdb_v2/http.go
@@ -19,6 +19,7 @@ import (
"strings"
"sync"
"sync/atomic"
+ "syscall"
"time"
"github.com/alitto/pond/v2"
@@ -328,6 +329,14 @@ func (c *httpClient) writeBatch(ctx context.Context, b *batch) error {
resp, err := c.client.Do(req.WithContext(ctx))
if err != nil {
internal.OnClientError(c.client, err)
+ if errors.Is(err, syscall.ECONNREFUSED) {
+ retryDuration := getRetryDuration(resp.Header, c.retryCount.Add(1))
+ return &ThrottleError{
+ Err: fmt.Errorf("failed to write to %s; will retry in %s. (%s)", b.bucket, retryDuration, resp.Status),
+ StatusCode: resp.StatusCode,
+ RetryAfter: retryDuration,
+ }
+ }
return err
}
defer resp.Body.Close()
```
Contributor guide
Research direction
Start in plugins/outputs/influxdb_v2/http.go at httpClient.writeBatch and compare connection-refused handling with the existing retry logic for HTTP status responses. Confirm that an ECONNREFUSED failure uses retryTime before another endpoint attempt, and verify the outputs.influxdb_v2 behavior with the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100