hashicorp / hashicorp/consul

Consul k8s sidecar proxies requests with scheme 'https' for h2c, which fails in .NET Core 3 gRPC service

Open
#7,199 3 comments 0 reactions 0 assignees View on GitHub
theme/health-checks type/enhancement waiting-reply
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

I would like to run my .NET gRPC service with a consul sidecar proxy within a pod on localhost without adding any TLS configuration. Therefore I want to make use the HTTP/2 clear-text (h2c) functionality.

After some debugging it seems however that the sidecar-proxy currently always proxies HTTP/2 requests with header "scheme" set to "https". Or perhaps it just takes the originating request, which would always be https too.

.NET core 3 does not allow this header when receiving a h2c request and will throw an error.

Also see this issue, in which majority seem to say it is by design: https://github.com/dotnet/aspnetcore/issues/14745

Is there some way of forcing the "scheme" header to "http" from sidecar to the microservice container within the pod?
If not, could this somehow be solved in the sidecar proxy?

#### Reproduction Steps

1. Create an AKS cluster
1. Deploy Consul k8s using Helm chart with `connectInject` enabled in the `values.yaml`
1. Deploy Ambassador Edge Stack and set-up TLS for `some.host.tld`
1. Deploy and configure Ambassador consul connector according to the documentation
1. Create a .NET core 3 gRPC service and set logging level to DEBUG
1. Deploy the gRPC service

gRPC service deploy yaml

```yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
generation: 60
labels:
service: my-grpc-service
name: my-grpc-service
namespace: default
spec:
replicas: 1
selector:
matchLabels:
service: my-grpc-service
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
annotations:
consul.hashicorp.com/connect-inject: "true"
consul.hashicorp.com/connect-service-port: "http"
labels:
service: my-grpc-service
spec:
containers:
image: my-grpc-service-image
imagePullPolicy: Always
name: my-grpc-service
ports:
- containerPort: 80
name: http
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
---
apiVersion: getambassador.io/v1
kind: Mapping
metadata:
name: my-grpc-service-mapping-tls
spec:
host: some.host.tld
prefix: /MyGrpcService/
rewrite: /MyGrpcService/
grpc: true
service: my-grpc-service-sidecar-proxy
resolver: consul-dc1
tls: ambassador-consul
load_balancer:
policy: round_robin
```

6. Generate gRPC code or use BloomRPC or similar tool to call a procedure
7. View the log of the .NET core gRPC service container
```
Microsoft.AspNetCore.Connections.ConnectionAbortedException: The request :scheme header 'https' does not match the transport scheme 'http'.
```

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 5
services = 5
build:
prerelease =
revision = 1200f25e
version = 1.6.2
consul:
acl = disabled
known_servers = 1
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 431
max_procs = 2
os = linux
version = go1.12.13
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5
members = 3
query_queue = 0
query_time = 1
```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 1200f25e
version = 1.6.2
consul:
acl = disabled
bootstrap = true
known_datacenters = 1
leader = true
leader_addr = 10.244.1.39:8300
server = true
raft:
applied_index = 713119
commit_index = 713119
fsm_pending = 0
last_contact = 0
last_log_index = 713119
last_log_term = 2
last_snapshot_index = 705997
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:7c5ee703-ab14-b2b8-3a24-5ff5cbbdcbb0 Address:10.244.1.39:8300}]
latest_configuration_index = 1
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 2
goroutines = 277
max_procs = 2
os = linux
version = go1.12.13
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```

### Operating system and Environment details

Azure AKS
Consul 1.6.2
Ambassador Edge Stack 1.0
Angular frontend using gRPC-Web
BloomRPC for testing

### Log Fragments
> I'm not sure how to set the log level in the k8s sidecar proxy

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.