hashicorp / hashicorp/nomad

Make Nomad's HTTP server KeepAlive settings configurable

Open
#15,567 0 comments 0 reactions 0 assignees View on GitHub
stage/accepted theme/api type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Why?
We are introducing a proxy in front of Nomad's Server HTTP API. As we load tested this new layer of our infrastructure we were bugged by constant and reproducible `EOF` errors and `read: connection reset by peer` errors when the load test was reaching its peak of ~68 concurrent requests.

Because this is an undocumented and hardcoded setting we went from tuning the TCP stack of our ECS cluster (where the proxy is running) all the way down to tcpdumps and straces trying to figure out why this was happening.

### TL;DR

https://github.com/hashicorp/nomad/blob/main/command/agent/http.go#L343 defines a default `keepalive` of 30s and our proxy has a 2 min default (we tuned it down to 60s during our debug but that didn't help).

This means that Nomad was closing the connection prematurely while our Proxy was considering it to still be alive. The healthcheck starts to get random 502's and mark the upstream as unhealthy.

When we set the `keepalive` value on our proxy config to `15s` we could not reproduce the issue anymore.

### Proposal
Can the Nomad HTTP server `keepalive` settings be configurable or at least match the [`wait` time default of `5m` defined for Blocking Queries](https://developer.hashicorp.com/nomad/api-docs#blocking-queries)? If not, can this be documented with a warning somewhere to save someone's else days of hair pulling?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in command/agent/http.go around line 343, where the 30s HTTP keepalive default is defined. Compare that setting with the 5m blocking-query wait documented in the Nomad API docs and inspect nearby HTTP server configuration and tests. Done means the keepalive behavior is configurable or the mismatch and proxy risk are clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.