hashicorp / hashicorp/consul

Consul connect proxy reset connection instead of timeout

Open
#21,877 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

I have grafana Loki running as a nomad service which gets connected to clients via consul proxy.

Main Runtime service

service {
name = "runtime"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "loki"
local_bind_port = 18092
}
}
}
}
}

Loki Service

service {
name = "loki"
port = 3100
check {
name = "Loki instance health check"
type = "http"
path = "/ready"
interval = "10s"
timeout = "2s"
expose = true
check_restart {
limit = 3
grace = "2m"
}
}
connect {
sidecar_service {}
sidecar_task {
resources {
cpu = 80
memory = 80
}
}
}
}

Everything works correctly, but when I stop the Loki service, I receive a 'connection reset' error from the Consul proxy, even after reducing the client connection timeout to 1 second (default connection timeout for the Consul proxy is 5 seconds). In this case, I expect an HTTP connection timeout instead of an immediate connection reset by the proxy.

# wget --timeout=1 http://127.0.0.1:18092/ready
Connecting to 127.0.0.1:18092 (127.0.0.1:18092)
wget: error getting response: Connection reset by peer

As you can see above, the Consul proxy reset the connection even though the client ("wget") has 1-second timeout.

Could someone please help with this? Am I missing some configuration in the nomad service spec?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.