Service Router filter uses incorrect metadata when a service has instances routed via terminating gateway
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
I have a service with instances directly available within the service mesh and externally registered. We are using the transparent proxy so those externally registered instances are reachable via the terminating gateway. This arrangement is because we are migrating services into the service mesh and updating them at the same time.
I want to use the service resolver to create a subset containing the externally-registered services and a second subset with those that have been migrated into the mesh. I added metadata to the external registration and expected
```
"Service.Meta.external == \"true\""
```
to pick them out but I consistently got no members in the envoy cluster of dependent services.
However I did get the right instances when (a random guess and I got lucky!) I used a filter expression of:
```
"Service.Meta[\"k8s-service-name\"] == \"consul-terminating-gateway\""
```
but although it worked on the source service, the subset on the terminating gateway itself was completely empty.
Combining both did yield the right result but is quite clunky and not very obvious!
```
"Service.Meta[\"k8s-service-name\"] == \"consul-terminating-gateway\" or Service.Meta.external == \"true\"
```
It appears that the endpoint metadata used by the filter is overwritten with the terminating gateway metadata, for external endpoints.
I believe the correct behaviour would be for the filter expression to use the metadata of the relevant service instance metadata itself and not the terminating gateway that the instance is exposed through.
---
#### Reproduction Steps
1. Provision a consul cluster with transparent proxy mode enabled and a terminating gateway instance.
2. Create a service A with one instance and intentions so that A can call service B. Give service A metadata kv "my-meta"="my-meta"
3. Create a service B with one instance.
4. Call the catalog api to register an external node
5. Call the catalog api to register an instance of service B against the node from step 4.
6. Check that the B shows in the terminating gateway and that the topology shows A linking to B
7. Create a service resolver:
```
kind: ServiceResolver
metadata:
name: service-A
spec:
defaultSubset: external
subsets:
external:
filter: "Service.Meta[\"k8s-service-name\"] == \"consul-terminating-gateway\""
onlyPassing: true
internal:
filter: "Service.Meta[\"my-meta\"] == \"my-meta\""
onlyPassing: true
```
8. Connect to A to see the envoy cluster config. Observe that the enoy cluster for B.external has 2 members
### Consul info for both Client and Server
EKS 1.28
Consul 1.17.3
Installed via helm chart
Transparent proxy enabled
Connect enabled
Contributor guide
Research direction
Start by reproducing the ServiceResolver configuration with transparent proxy mode and a terminating gateway, then inspect the service resolver filtering path and endpoint metadata handling. Compare the metadata from the external service instance with the terminating gateway metadata while checking the Envoy cluster configuration. Done means filters select instances using the relevant service instance metadata and both subsets populate as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100