aws / aws/aws-app-mesh-roadmap
Feature Request: Improve documentation around Host Matching of Virtual Gateways
- Dominant language
- No language data
- Stars
- 344
- Forks
- 24
- PR merge metrics
- No merged PRs in 30d
Description
*If you want to see App Mesh implement this idea, please upvote with a :+1:.*
**Tell us about your request**
Improve documentation around Host Matching of Virtual Gateways.
**Which integration(s) is this request for?**
N/A
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
We're deploying Virtual Gateway using ECS Fargate. The Virtual Gateway serves as a single ingress to our AppMesh, handling ingress from multiple sources:
- API GW + NLB
- ALB with multiple domains registered for it.
To this point, we only used the API GW + NLB, and terraform `aws_appmesh_gateway_resource`, which lacks support for host matching (see: https://github.com/hashicorp/terraform-provider-aws/issues/20272). As a result, we had ~30 routes on Virtual Gateway, and they only used `prefix` matching.
Once we added ALB, we wanted to:
1. Update the existing routes to only apply for the traffic from NLB by adding `host/exact` match to them
2. Create new routes with `host/exact` match to properly route traffic from any of the Domains registered for ALB.
We wanted to approach pt. 1. gradually, with every one of ~30 service updating the routes at its own pace (we have separate CICD pipelines for each service, so this would take a week or two, depending on their rollout process).
To a great surprise, however, adding a single rule with `host/exact` match on the NLB Domain, stopped all other rules from working.
This is, because, VG Route matches are not evaluated on per-route basis. Instead - and what the documentation does not clearly explain - Envoy groups the routes into `VirtualHosts`, and only then adds `prefix` matches to them. This way, Envoy **first** matches on the Host Name, and **only then** looks for a `prefix` match.
Look at this (simplified) output of `/config_dump` on our Virtual Gateway Envoy
1. Initial configuration:
Our Routes only used `prefix` match, and ended up in a `wildcard` Virtual Host. All routes were matching requests and worked correctly.
```yaml
virtual_hosts:
- name: *
domains:
- *
routes:
- match: { "prefix": "/service-a"}
route: { "host_rewrite_literal": "service-a.dev.local"}
- match: { "prefix": "/service-b"}
route: { "host_rewrite_literal": "service-b.dev.local"}
- match: { "prefix": "/service-c"}
route: { "host_rewrite_literal": "service-c.dev.local"}
```
2. Adding routes that match the ALB HostName, while the NLB still does not use `host/exact`:
You can see how all "old" routes still remain in the `wildcard` Virtual Host. The new Route one only matches ALB Host, so everything still works.
```yaml
virtual_hosts:
- name: * # This - by default - handles existing traffic from
domains: # API GW and NLB:
- * # nlb-1111111111111111.elb.eu-west-1.amazonaws.com
routes:
- match: { "prefix": "/service-a"}
route: { "host_rewrite_literal": "service-a.dev.local"}
- match: { "prefix": "/service-b"}
route: { "host_rewrite_literal": "service-b.dev.local"}
- match: { "prefix": "/service-c"}
route: { "host_rewrite_literal": "service-c.dev.local"}
- name: myapp.dev-atech.com # This domain points to ALB at:
domains: # alb-0000000000000000.elb.eu-west-1.amazonaws.com
- myapp.dev-atech.com
myapp.dev-atech.com:8080
routes:
- match: { "prefix": "/kafka-ui"}
route: { "host_rewrite_literal": "myapp.dev.local"}
```
There's a risk, however. With this configuration, an attacker can hit the IP of the ALB with a fake `Host: foo.com` header and this will trigger the `wildcard` Virtual Host. An attacker can then access the services that should only be accessible via NLB, using our ALB. That's wrong.
3. Specifying `host/exact` match on "old" NLB Routes
This brings us to the clue of this ticket. Gradually specifying `host/exact` matches on NLB Routes. When we did that for just a single service, our configuration turned into this:
```yaml
virtual_hosts:
- name: *
domains:
- *
routes:
- match: { "prefix": "/service-a"} # None of these will match if traffic originates from
route: { "host_rewrite_literal": "service-a.dev.local"} # nlb-1111111111111111.elb.eu-west-1.amazonaws.com
# because the host match will point to VirtualHost below!
- match: { "prefix": "/service-b"}
route: { "host_rewrite_literal": "service-b.dev.local"}
- name: nlb-1111111111111111.elb.eu-west-1.amazonaws.com # <<< This breaks all above!
domains: # That's because the "domain" name match is done first,
- nlb-1111111111111111.elb.eu-west-1.amazonaws.com # and the "prefix" is matched after the "Virtual Host"
nlb-1111111111111111.elb.eu-west-1.amazonaws.com:8080 # is already selected. So all traffix that originates
routes: # from: nlb-1111111111111111.elb.eu-west-1.amazonaws.com
- match: { "prefix": "/service-c"} # will first pick VirtualHost, and only then try to match
route: { "host_rewrite_literal": "service-c.dev.local"} # the - now very short - list of path prefixes.
- name: myapp.dev-atech.com
domains:
- myapp.dev-atech.com
myapp.dev-atech.com:8080
routes:
- match: { "prefix": "/"}
route: { "host_rewrite_literal": "myapp.dev.local"}
```
Updating `service-c` Rule to use `host/exact` created new Envoy Virtual Host, which now captures all requests from NLB, and only after they're assigned to that Virtual Host, `prefix` matching is performed. As a result, Rules for `service-a` and `service-b` stopped matching immediately after the Rule for `service-c` was updated.
**Are you currently working around this issue?**
We're deploying separate Virtual Gateways for each ingress. This simplifies the configuration and eradicates the risks mentioned in pt.2. We're also a bit more educated about how Envoy configuration really looks like, and that it's not reflecting 1-to-1 the abstraction that AppMesh VirtualGateway / VirtualGatewayRoute configuration is.
It would be good, though, if the AppMesh Documentation on Gateway Routes(https://docs.aws.amazon.com/app-mesh/latest/userguide/gateway-routes.html) was more explicit on how this matching really works, and especially on the fact, that adding a single `host/exact` Rule to an existing Virtual Gateway may break all other rules if they don't specify `host` matcher.
Contributor guide
Research direction
Start with the Gateway Routes documentation at https://docs.aws.amazon.com/app-mesh/latest/userguide/gateway-routes.html and compare its host matching guidance with the reported Envoy /config_dump examples. Done means the page clearly explains VirtualHost selection, host and prefix matching, and the risk of adding a single host/exact route to an existing Virtual Gateway.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, terraform
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100