influxdata / influxdata/telegraf
Handle HTTP/2 GOAWAY messages when writing to outputs
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Use Case
HTTP/2 introduced a means for servers to tell clients to stop sending requests over a specific connection - the GOAWAY signal - (HTTP/1.1 needed to wait until it finished processing a request and then send `Connection: Close`, HTTP/2 does not).
AWS's Application Load Balancers (ALBs) sometimes send GOAWAY messages, for various reasons, the two main documented ones being
- if the compressed length of any of the headers exceeds 8 K bytes
- if the number of requests served through one connection exceeds 10,000
So, if an output connection is sufficiently long-lived that it carries 10,000 writes (which may not be that long a time for a busy instance), the ALB will eventually send a GOAWAY
### Expected behavior
Telegraf should receive the `GOAWAY`, close the connection, and resubmit over a new one.
### Actual behavior
Telegraf logs an error
```
2022-09-12T10:37:00Z E! [outputs.influxdb] When writing to [https://[ALB address]:8086/]: failed doing req: Post "https://[ALB address]:8086/write?db=messaging": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
2022-09-12T10:37:00Z E! [agent] Error writing to outputs.influxdb: could not write any address
```
### Additional info
The data isn't lost - because the write didn't complete successfully, it remains in the buffer and will be written out at the next flush interval.
But, it does lead to a level of log noise where telegraf's being used as an aggregator and performing a lot of writes.
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by tracing Telegraf's output write path and its HTTP/2 request handling, then reproduce the GOAWAY error if possible. Done means a GOAWAY closes the connection and the write is resubmitted over a new connection without the reported log error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100