hashicorp / hashicorp/consul

Connect / Envoy : 503 error "upstream connect error or disconnect/reset before headers" if h2c not enabled

Open
#6,657 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

Envoy seems to honor the header `Upgrade: h2` **even if the Consul service is configured with the `http` protocol.**
This can lead to unexpected 503 errors (with message `upstream connect error or disconnect/reset before header`) when making multiple HTTP request reusing the same connection **if the upstream service doesn't support `h2c`.**

Our use case is:
* We had some legacy LAMP stacks connected directly to the Internet on public IPs. The Apache servers was configured globally to enable `h2` only and not `h2c` (because most clients won't upgrade to `h2c` anyway), even if TLS wasn't enabled on the vHosts
* We want to move those stacks to private IPs and adds some edge proxies to balance the incoming traffic through Consul Connect with Envoy
* Because TLS is not enabled on those vHosts, the Consul services are configured with `protocol = http`

When a client is trying to access an application:
1. Client sends a HTTP/1.1 request on port 80 (i.e without TLS) to the edge proxy which is routed back to the application through Envoy proxies
2. The Apache server send the HTTP response with an `Upgrade: h2` header (seems to be the default behavior in Apache even if TLS is not enabled on the vHost)
3. Because it talks to the edge proxy without TLS, the client ignores the `Upgrade: h2` header and sends another HTTP/1.1 request reusing the same connection
4. **Because the downstream Envoy proxy saw the header `Upgrade: h2` on the first response, and because TLS is enabled on its connection to the upstream Envoy proxy, it upgrades the request to HTTP2**
5. The upstream proxy delivers the upgraded HTTP2 request to the Apache server but without TLS. Because `h2c` isn't enabled on the Apache server, it drops the request
6. Client get a 503 error with `upstream connect error or disconnect/reset before headers`

I'm not sure if this is really a bug, but this behavior is quite confusing and maybe it should be documented.
One major use case for Connect is precisely to seamlessly add TLS security when connecting applications (i.e without modifying the applications configurations), which isn't the case here.
In addition, because the services was configured with `protocol = http` I think most users may expect this parameter to force Envoy to use HTTP/1.1.

#### Reproduction Steps

Steps to reproduce this issue:

1. Launch this Docker-compose test setup:

consul-envoy Dockerfile

```dockerfile
FROM consul:latest
FROM envoyproxy/envoy:v1.11.2
COPY --from=0 /bin/consul /bin/consul
ENTRYPOINT ["consul", "connect", "envoy"]
```

consul.hcl

```hcl
enable_central_service_config = true

services {
name = "reverse-proxy"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "web"
local_bind_port = 80
}
}
}
}
}

services {
name = "web"
port = 8080
connect {
sidecar_service {
proxy {}
}
}
}

config_entries {
bootstrap {
kind = "service-defaults"
name = "web"
protocol = "http"
}
}
```

httpd.conf

```apache
ServerRoot "/usr/local/apache2"

Listen 8080
DocumentRoot "/usr/local/apache2/htdocs"

LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule dir_module modules/mod_dir.so
LoadModule http2_module modules/mod_http2.so

User daemon
Group daemon

DirectoryIndex index.html

ErrorLog /proc/self/fd/2
LogLevel warn
LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog /proc/self/fd/1 common

Protocols h2 http/1.1
```

docker-compose.yml

```yaml
version: '3.7'

services:
web:
image: httpd:2.4.41-alpine
network_mode: "host"
volumes:
- ./files/httpd.conf:/usr/local/apache2/conf/httpd.conf:ro

consul:
image: consul:1.6.1
command: "agent -dev -config-file /etc/consul/consul.hcl"
network_mode: "host"
volumes:
- ./files/consul.hcl:/etc/consul/consul.hcl:ro

sidecar_reverse_proxy:
image: consul-envoy
command: "-sidecar-for reverse-proxy -admin-bind localhost:19000 -- -l debug"
network_mode: "host"
depends_on:
- consul

sidecar_web:
image: consul-envoy
command: "-sidecar-for web -admin-bind localhost:19001 -- -l debug"
network_mode: "host"
depends_on:
- consul
```

2. Run `curl -v http://127.0.0.1 --next http://127.0.0.1`
3. The second request fails with:

```
< HTTP/1.1 503 Service Unavailable
< content-length: 95
< content-type: text/plain
< date: Sat, 19 Oct 2019 09:10:51 GMT
< server: envoy
<
* Connection #0 to host 127.0.0.1 left intact
upstream connect error or disconnect/reset before headers. reset reason: connection termination
```

### Operating system and Environment details

Consul 1.6.0 or 1.6.1
Envoy 1.8.0, 1.9.1, 1.10.0, 1.11.1 or 1.11.2
Apache 2.4.41
Docker 19.03.4 build 9013bf583a
Ubuntu 18.04

### Log Fragments

[Sidecars logs](https://gist.github.com/jrrdev/156dee13ec47494eb6a03be26ebd9d4b)

curl verbose output

```
curl -v http://127.0.0.1 --next http://127.0.0.1

* Rebuilt URL to: http://127.0.0.1/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< date: Sat, 19 Oct 2019 09:10:45 GMT
< server: envoy
< upgrade: h2
< last-modified: Mon, 11 Jun 2007 18:53:14 GMT
< etag: "2d-432a5e4a73a80"
< accept-ranges: bytes
< content-length: 45
< x-envoy-upstream-service-time: 8
<

It works!


* Connection #0 to host 127.0.0.1 left intact
* Rebuilt URL to: http://127.0.0.1/
* Found bundle for host 127.0.0.1: 0x55f37b6a8a50 [can pipeline]
* Re-using existing connection! (#0) with host 127.0.0.1
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 95
< content-type: text/plain
< date: Sat, 19 Oct 2019 09:10:51 GMT
< server: envoy
<
* Connection #0 to host 127.0.0.1 left intact
upstream connect error or disconnect/reset before headers. reset reason: connection termination
```

Contributor guide

Open the contributing guide

Research direction

Start with docker-compose.yml, consul.hcl, httpd.conf, and the reproduction command `curl -v http://127.0.0.1 --next http://127.0.0.1`. Compare the first and reused requests with the supplied Envoy logs to determine whether the reported protocol upgrade is expected. Done means establishing the behavior and documenting or correcting it, with the two-request reproduction no longer producing an unexplained 503.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.