Make Nomad's HTTP server KeepAlive settings configurable
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Why?
We are introducing a proxy in front of Nomad's Server HTTP API. As we load tested this new layer of our infrastructure we were bugged by constant and reproducible `EOF` errors and `read: connection reset by peer` errors when the load test was reaching its peak of ~68 concurrent requests.
Because this is an undocumented and hardcoded setting we went from tuning the TCP stack of our ECS cluster (where the proxy is running) all the way down to tcpdumps and straces trying to figure out why this was happening.
### TL;DR
https://github.com/hashicorp/nomad/blob/main/command/agent/http.go#L343 defines a default `keepalive` of 30s and our proxy has a 2 min default (we tuned it down to 60s during our debug but that didn't help).
This means that Nomad was closing the connection prematurely while our Proxy was considering it to still be alive. The healthcheck starts to get random 502's and mark the upstream as unhealthy.
When we set the `keepalive` value on our proxy config to `15s` we could not reproduce the issue anymore.
### Proposal
Can the Nomad HTTP server `keepalive` settings be configurable or at least match the [`wait` time default of `5m` defined for Blocking Queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries)? If not, can this be documented with a warning somewhere to save someone's else days of hair pulling?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in command/agent/http.go around line 343, where the 30s HTTP keepalive default is defined. Compare that setting with the 5m blocking-query wait documented in the Nomad API docs and inspect nearby HTTP server configuration and tests. Done means the keepalive behavior is configurable or the mismatch and proxy risk are clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100