juju / juju/juju-controller

metrics-endpoint-relation-created hook fails with ValueError on k8s when bind_address returns service FQDN

Open
#130 0 comments 0 reactions 1 assignee View on GitHub

@gfouillet is already working on this.

Since Aug 24, 2026.

Dominant language
Python
Stars
3
Forks
19
Avg merge
1d 15h
Merged PRs (30d)
8

Description

Symptom

On a k8s controller, integrating juju-controller with prometheus-k8s via
controller:metrics-endpoint causes the metrics-endpoint-relation-created
hook to fail repeatedly with:

Traceback (most recent call last):
  File ".../charms/prometheus_k8s/v0/prometheus_scrape.py", line 1523, in set_scrape_job_spec
    self._set_unit_ip()
  File ".../prometheus_scrape.py", line 1554, in _set_unit_ip
    unit_ip = str(self._charm.model.get_binding(relation).network.bind_address)
  File ".../ops/model.py", line 1084, in _network_get
    return Network(self._backend.network_get(name, relation_id))
  File ".../ops/model.py", line 1230, in NetworkInterface.__init__
    subnet = ipaddress.ip_network(address)
  File "/usr/lib/python3.12/ipaddress.py", line 83, in ip_network
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 network')
ValueError: 'controller-0.controller-service-endpoints.controller-ctrl-<id>.svc.cluster.local'
          does not appear to be an IPv4 or IPv6 network

Followed immediately by:

WARNING unit.controller/0.juju-log metrics-endpoint:1:
  0 containers are present in metadata.yaml and refresh_event was not specified.
  Defaulting to update_status. Metrics IP may not be set in a timely fashion.

The uniter retries every ~30 s for ~13 minutes; SAAS offer stays in error;
check_prometheus_targets exhausts its 30-retry budget.

Reproduction (CI gating)

https://jenkins.juju.canonical.com/job/test-controllercharm-test-prometheus-microk8s/,
builds #3703 #3706 #3707 #3708 #3724 #3725 (all 4.1-beta2.x / main /
feature/controller-snap):

juju offer controller.controller:metrics-endpoint
juju deploy prometheus-k8s --channel 1/stable --trust
juju relate prometheus-k8s admin/controller.controller
# controller SAAS → 'error' after ~5s
# after 12m50s:  check_prometheus_targets prometheus-k8s 0 failed after 30 retries

Charm revs observed failing: juju-controller-157 (4.1-beta2 main),
juju-controller-328 (4.0.15.x). 3.6.28 / juju-controller-361 does not
exhibit the failure.

Tracked from juju side at https://warthogs.atlassian.net/browse/JUJU-10266.

Root cause

lib/charms/prometheus_k8s/v0/prometheus_scrape.py:_set_unit_ip calls

unit_ip = str(self._charm.model.get_binding(relation).network.bind_address)

binding.network.bind_address accesses ops.model.NetworkInterface.__init__,
which calls ipaddress.ip_network(address). When the controller pod has just
been created on k8s, network_get returns the service-endpoints DNS FQDN
(<unit>.controller-service-endpoints.<ns>.svc.cluster.local, matching the
ControllerServiceFQDNTemplate in juju/internal/provider/kubernetes/constants/constants.go),
which is not a valid CIDR. The ValueError propagates as an uncaught
exception, the hook fails, and because the charm has no containers: and no
refresh_event, the IP is not re-evaluated — every ~30 s retry repeats
the same race.

Note: the same _set_unit_ip line exists in the upstream main at
revision 58 of prometheus_k8s/v0/prometheus_scrape.py, with the same
missing guard.

Suggested fixes

Either of:

  1. Charmlib — wrap the bind_address access in a try/except and defer
    the relation-data write via a deferred event handler when the binding is
    unresolved; or generally wrap set_scrape_job_spec so that _set_unit_ip
    failures do not fail the hook.

  2. Charm — add a containers: stanza with refresh_event: <container>-pebble-ready
    in metadata.yaml and observe PebbleReady for metrics-endpoint. This
    also suppresses the "0 containers in metadata.yaml" warning and gives
    the IP a retry path through the operator framework.

Local mitigation until either lands

In juju/tests/suites/controllercharm/prometheus.sh, after juju relate,
add a wait_for "controller SAAS status == active" before
check_prometheus_targets, so failures surface with the actual error rather
than a 12m50s retry timeout (see https://warthogs.atlassian.net/browse/JUJU-10266).

Likely related
  • #93 "Wrong host address for metrics scrape job" — same _set_unit_ip
    line (main @ line 1692 / rev-157 @ line 1554) returns a non-IP target
    without crashing. I believe #93 and this issue share a root cause:
    missing guard on bind_address and no retry path when the binding is
    not yet IP-resolvable.
  • #112 "metrics-endpoint-relation-created on every unit fail in cluster
    mode" — different failure (APIError: user "juju-metrics-r1" already exists) on the same hook, but a third race condition on the relation
    lifecycle.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.