Traefik route conflicts between otel and tempo in the COS stack

In the COS stack, both tempo and otelcol request the same listener ports / entry points from traefik. And they request trafik that traffic from these ports to be routed to their own backend service. For example look at the entry points in traefiks static configuration

```yaml
entryPoints:
diagnostics:
address: :8082
health:
address: :13133
jaeger-grpc:
address: :14250
jaeger-thrift-http:
address: :14268
loki-http:
address: :3500
metrics:
address: :8888
otlp-grpc:
address: :4317
otlp-http:
address: :4318
tempo-grpc:
address: :9096
tempo-http:
address: :3200
web:
address: :80
websecure:
address: :443
zipkin:
address: :9411
global:
checknewversion: false
log:
level: DEBUG
metrics:
prometheus:
addRoutersLabels: true
addServicesLabels: true
entryPoint: diagnostics
ping:
entryPoint: diagnostics
providers:
file:
directory: /opt/traefik/juju
watch: true
```

The following entry points are common between both tempo and otelcol

```yaml
jaeger-grpc:
address: :14250
jaeger-thrift-http:
address: :14268
otlp-grpc:
address: :4317
otlp-http:
address: :4318
zipkin:
address: :9411
```

and the look at the dynamic configs submitted by both the apps. otelcol first:

```yaml
http:
routers:
juju-cos-dev-otelcol-health:
entryPoints:
- health
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-health
juju-cos-dev-otelcol-health-tls:
entryPoints:
- health
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-health
tls: {}
juju-cos-dev-otelcol-jaeger-grpc:
entryPoints:
- jaeger-grpc
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-jaeger-grpc
juju-cos-dev-otelcol-jaeger-grpc-tls:
entryPoints:
- jaeger-grpc
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-jaeger-grpc
tls: {}
juju-cos-dev-otelcol-jaeger-thrift-http:
entryPoints:
- jaeger-thrift-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-jaeger-thrift-http
juju-cos-dev-otelcol-jaeger-thrift-http-tls:
entryPoints:
- jaeger-thrift-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-jaeger-thrift-http
tls: {}
juju-cos-dev-otelcol-loki-http:
entryPoints:
- loki-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-loki-http
juju-cos-dev-otelcol-loki-http-tls:
entryPoints:
- loki-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-loki-http
tls: {}
juju-cos-dev-otelcol-metrics:
entryPoints:
- metrics
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-metrics
juju-cos-dev-otelcol-metrics-tls:
entryPoints:
- metrics
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-metrics
tls: {}
juju-cos-dev-otelcol-otlp-grpc:
entryPoints:
- otlp-grpc
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-otlp-grpc
juju-cos-dev-otelcol-otlp-grpc-tls:
entryPoints:
- otlp-grpc
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-otlp-grpc
tls: {}
juju-cos-dev-otelcol-otlp-http:
entryPoints:
- otlp-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-otlp-http
juju-cos-dev-otelcol-otlp-http-tls:
entryPoints:
- otlp-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-otlp-http
tls: {}
juju-cos-dev-otelcol-zipkin:
entryPoints:
- zipkin
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-zipkin
juju-cos-dev-otelcol-zipkin-tls:
entryPoints:
- zipkin
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-otelcol-service-zipkin
tls: {}
services:
juju-cos-dev-otelcol-service-health:
loadBalancer:
servers:
- url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:13133
juju-cos-dev-otelcol-service-jaeger-grpc:
loadBalancer:
servers:
- url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:14250
juju-cos-dev-otelcol-service-jaeger-thrift-http:
loadBalancer:
servers:
- url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:14268
juju-cos-dev-otelcol-service-loki-http:
loadBalancer:
servers:
- url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:3500
juju-cos-dev-otelcol-service-metrics:
loadBalancer:
servers:
- url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:8888
juju-cos-dev-otelcol-service-otlp-grpc:
loadBalancer:
servers:
- url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:4317
juju-cos-dev-otelcol-service-otlp-http:
loadBalancer:
servers:
- url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:4318
juju-cos-dev-otelcol-service-zipkin:
loadBalancer:
servers:
- url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:9411
```

This says, for example, the `zipkin` entrypoint should be forwarded to the backend service `juju-cos-dev-otelcol-service-zipkin` ie to the address `http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:9411`.

Lets look at the dynamic config requested by tempo now,

```yaml
http:
routers:
juju-cos-dev-tempo-jaeger-grpc:
entryPoints:
- jaeger-grpc
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-jaeger-grpc
juju-cos-dev-tempo-jaeger-grpc-tls:
entryPoints:
- jaeger-grpc
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-jaeger-grpc
tls: {}
juju-cos-dev-tempo-jaeger-thrift-http:
entryPoints:
- jaeger-thrift-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-jaeger-thrift-http
juju-cos-dev-tempo-jaeger-thrift-http-tls:
entryPoints:
- jaeger-thrift-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-jaeger-thrift-http
tls: {}
juju-cos-dev-tempo-otlp-grpc:
entryPoints:
- otlp-grpc
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-otlp-grpc
juju-cos-dev-tempo-otlp-grpc-tls:
entryPoints:
- otlp-grpc
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-otlp-grpc
tls: {}
juju-cos-dev-tempo-otlp-http:
entryPoints:
- otlp-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-otlp-http
juju-cos-dev-tempo-otlp-http-tls:
entryPoints:
- otlp-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-otlp-http
tls: {}
juju-cos-dev-tempo-tempo-grpc:
entryPoints:
- tempo-grpc
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-tempo-grpc
juju-cos-dev-tempo-tempo-grpc-tls:
entryPoints:
- tempo-grpc
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-tempo-grpc
tls: {}
juju-cos-dev-tempo-tempo-http:
entryPoints:
- tempo-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-tempo-http
juju-cos-dev-tempo-tempo-http-tls:
entryPoints:
- tempo-http
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-tempo-http
tls: {}
juju-cos-dev-tempo-zipkin:
entryPoints:
- zipkin
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-zipkin
juju-cos-dev-tempo-zipkin-tls:
entryPoints:
- zipkin
rule: ClientIP(`0.0.0.0/0`)
service: juju-cos-dev-tempo-service-zipkin
tls: {}
services:
juju-cos-dev-tempo-service-jaeger-grpc:
loadBalancer:
servers:
- url: h2c://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:14250
juju-cos-dev-tempo-service-jaeger-thrift-http:
loadBalancer:
servers:
- url: http://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:14268
juju-cos-dev-tempo-service-otlp-grpc:
loadBalancer:
servers:
- url: h2c://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:4317
juju-cos-dev-tempo-service-otlp-http:
loadBalancer:
servers:
- url: http://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:4318
juju-cos-dev-tempo-service-tempo-grpc:
loadBalancer:
servers:
- url: h2c://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:9096
juju-cos-dev-tempo-service-tempo-http:
loadBalancer:
servers:
- url: http://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:3200
juju-cos-dev-tempo-service-zipkin:
loadBalancer:
servers:
- url: http://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:9411
```

Now this says, the same entrypoint `zipkin` should be forwarded to the backend service `juju-cos-dev-tempo-service-zipkin` ie to the address `http://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:9411`.

There are 5 such conflcts in the COS stacks default deployment. So which backend is actually being used? In my tests it was tempo. how? why? I dont know because traefik does not document this behavior which is weird. But its also weird to wire the same entry point to different backends without any path prefix and expect sane results. This feels like a critical network topology hole in the architecture which would need to be addressed urgently.

In the `istio-ingress-k8s` charm, we handle this by detecting such conflicts and putting the ingress charm in a blocked state with an appropriate message and logs and ingress is provided to neither of the conflicting charms. But traefik silently lets the faulty configuration to cascade.