Consul k8s sidecar proxies requests with scheme 'https' for h2c, which fails in .NET Core 3 gRPC service
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
I would like to run my .NET gRPC service with a consul sidecar proxy within a pod on localhost without adding any TLS configuration. Therefore I want to make use the HTTP/2 clear-text (h2c) functionality.
After some debugging it seems however that the sidecar-proxy currently always proxies HTTP/2 requests with header "scheme" set to "https". Or perhaps it just takes the originating request, which would always be https too.
.NET core 3 does not allow this header when receiving a h2c request and will throw an error.
Also see this issue, in which majority seem to say it is by design: https://github.com/dotnet/aspnetcore/issues/14745
Is there some way of forcing the "scheme" header to "http" from sidecar to the microservice container within the pod?
If not, could this somehow be solved in the sidecar proxy?
#### Reproduction Steps
1. Create an AKS cluster
1. Deploy Consul k8s using Helm chart with `connectInject` enabled in the `values.yaml`
1. Deploy Ambassador Edge Stack and set-up TLS for `some.host.tld`
1. Deploy and configure Ambassador consul connector according to the documentation
1. Create a .NET core 3 gRPC service and set logging level to DEBUG
1. Deploy the gRPC service
gRPC service deploy yaml
```yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
generation: 60
labels:
service: my-grpc-service
name: my-grpc-service
namespace: default
spec:
replicas: 1
selector:
matchLabels:
service: my-grpc-service
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
annotations:
consul.hashicorp.com/connect-inject: "true"
consul.hashicorp.com/connect-service-port: "http"
labels:
service: my-grpc-service
spec:
containers:
image: my-grpc-service-image
imagePullPolicy: Always
name: my-grpc-service
ports:
- containerPort: 80
name: http
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
---
apiVersion: getambassador.io/v1
kind: Mapping
metadata:
name: my-grpc-service-mapping-tls
spec:
host: some.host.tld
prefix: /MyGrpcService/
rewrite: /MyGrpcService/
grpc: true
service: my-grpc-service-sidecar-proxy
resolver: consul-dc1
tls: ambassador-consul
load_balancer:
policy: round_robin
```
6. Generate gRPC code or use BloomRPC or similar tool to call a procedure
7. View the log of the .NET core gRPC service container
```
Microsoft.AspNetCore.Connections.ConnectionAbortedException: The request :scheme header 'https' does not match the transport scheme 'http'.
```
### Consul info for both Client and Server
Client info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 5
services = 5
build:
prerelease =
revision = 1200f25e
version = 1.6.2
consul:
acl = disabled
known_servers = 1
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 431
max_procs = 2
os = linux
version = go1.12.13
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5
members = 3
query_queue = 0
query_time = 1
```
Server info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 1200f25e
version = 1.6.2
consul:
acl = disabled
bootstrap = true
known_datacenters = 1
leader = true
leader_addr = 10.244.1.39:8300
server = true
raft:
applied_index = 713119
commit_index = 713119
fsm_pending = 0
last_contact = 0
last_log_index = 713119
last_log_term = 2
last_snapshot_index = 705997
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:7c5ee703-ab14-b2b8-3a24-5ff5cbbdcbb0 Address:10.244.1.39:8300}]
latest_configuration_index = 1
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 2
goroutines = 277
max_procs = 2
os = linux
version = go1.12.13
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```
### Operating system and Environment details
Azure AKS
Consul 1.6.2
Ambassador Edge Stack 1.0
Angular frontend using gRPC-Web
BloomRPC for testing
### Log Fragments
> I'm not sure how to set the log level in the k8s sidecar proxy
Contributor guide
Assessment
This issue has not been assessed yet.