Wrong host address for metrics scrape job

Hello,

I am trying to scrape the metrics of my Juju controller charm using Opentelemetry Collector (inside my controller model) and then remote write them to a Cos Lite instance over a CMR. The problem is, when I look at what the controller has written to the relation databag with Otelcol, it seems like the prometheus_scrape_unit_address is not reachable by the collector. In fact, that host doesn't seem to exist on my machine.

Here is my status:

Model       Controller  Cloud/Region  Version  SLA          Timestamp
controller  ck8s        ck8s          3.6.11   unsupported  17:31:13-05:00

SAAS  Status  Store  URL
prom  active  ck8s   admin/cos-lite.prom

App         Version  Status  Scale  Charm                        Channel     Rev  Address        Exposed  Message
controller           active      1  juju-controller              3.6/stable  116                 yes      
otel        0.130.1  active      1  opentelemetry-collector-k8s                2  10.152.183.77  no       

Unit           Workload  Agent  Address     Ports      Message
controller/0*  active    idle   10.1.0.71   37017/TCP  
otel/0*        active    idle   10.1.0.184                       

Offer       Application  Charm            Rev  Connected  Endpoint          Interface          Role
controller  controller   juju-controller  116  1/1        metrics-endpoint  prometheus_scrape  provider

Integration provider         Requirer                   Interface                Type     Message
controller:metrics-endpoint  otel:metrics-endpoint      prometheus_scrape        regular  
prom:receive-remote-write    otel:send-remote-write     prometheus_remote_write  regular  

The unit IP for the controller, as you can see, is 10.1.0.71. I expect to see the same IP as a scrape target in the Otelcol config. However, in the config, the relevant config block shows:

prometheus/metrics-endpoint/otel/0:
    config:
      scrape_configs:
      - basic_auth:
          password: foobar
          username: foobar
        job_name: juju_controller_57d71968_controller_prometheus_scrape-0
        metrics_path: /introspection/metrics
        relabel_configs:
        - regex: (.*)
          separator: _
          source_labels:
          - juju_model
          - juju_model_uuid
          - juju_application
          - juju_unit
          target_label: instance
        scheme: https
        scrape_interval: 1m
        scrape_timeout: 10s
        static_configs:
        - labels:
            juju_application: controller
            juju_charm: juju-controller
            juju_model: controller
            juju_model_uuid: 57d71968-7396-4b74-8afb-c935e3770802
            juju_unit: controller/0
          targets:
          - 10.1.0.241:17070
        tls_config:
          ca_file: /etc/ssl/certs/otel_juju_controller_57d71968_controller_prometheus_scrape_0_ca.pem
          insecure_skip_verify: false
          server_name: juju-apiserver

Otelcol fails to scrape the controller because 10.1.0.241 is not actually a valid target. If I retry with 10.1.0.71 , it works.
The relevant LOC seems to be
https://github.com/juju/juju-controller/blob/abb5f63c279ea3a53f032650c09c65a5fe0cb592/src/charm.py#L87-L106

10.1.0.241 seems to be coming from
https://github.com/juju/juju-controller/blob/abb5f63c279ea3a53f032650c09c65a5fe0cb592/lib/charms/prometheus_k8s/v0/prometheus_scrape.py#L1543-L1567

but why is the bind address not reachable?

I'd appreciate any thoughts or ideas here.

Why is the unit IP passed here and instead a wild card is passed?

Related issues:

• https://github.com/canonical/opentelemetry-collector-k8s-operator/issues/132

Start with src/charm.py lines 87-106 and lib/charms/prometheus_k8s/v0/prometheus_scrape.py lines 1543-1567, then compare the generated target 10.1.0.241 with the unit address 10.1.0.71 in the provided OpenTelemetry configuration. Determine why the advertised address is unreachable and confirm that the resulting target can scrape /introspection/metrics through Otelcol.