metrics-endpoint-relation-created hook fails with ValueError on k8s when bind_address returns service FQDN
@gfouillet is already working on this.
Since Aug 24, 2026.
- Dominant language
- Python
- Stars
- 3
- Forks
- 19
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 8
Description
Symptom
On a k8s controller, integrating juju-controller with prometheus-k8s via
controller:metrics-endpoint causes the metrics-endpoint-relation-created
hook to fail repeatedly with:
Traceback (most recent call last):
File ".../charms/prometheus_k8s/v0/prometheus_scrape.py", line 1523, in set_scrape_job_spec
self._set_unit_ip()
File ".../prometheus_scrape.py", line 1554, in _set_unit_ip
unit_ip = str(self._charm.model.get_binding(relation).network.bind_address)
File ".../ops/model.py", line 1084, in _network_get
return Network(self._backend.network_get(name, relation_id))
File ".../ops/model.py", line 1230, in NetworkInterface.__init__
subnet = ipaddress.ip_network(address)
File "/usr/lib/python3.12/ipaddress.py", line 83, in ip_network
raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 network')
ValueError: 'controller-0.controller-service-endpoints.controller-ctrl-<id>.svc.cluster.local'
does not appear to be an IPv4 or IPv6 network
Followed immediately by:
WARNING unit.controller/0.juju-log metrics-endpoint:1:
0 containers are present in metadata.yaml and refresh_event was not specified.
Defaulting to update_status. Metrics IP may not be set in a timely fashion.
The uniter retries every ~30 s for ~13 minutes; SAAS offer stays in error;
check_prometheus_targets exhausts its 30-retry budget.
Reproduction (CI gating)
https://jenkins.juju.canonical.com/job/test-controllercharm-test-prometheus-microk8s/,
builds #3703 #3706 #3707 #3708 #3724 #3725 (all 4.1-beta2.x / main /
feature/controller-snap):
juju offer controller.controller:metrics-endpoint
juju deploy prometheus-k8s --channel 1/stable --trust
juju relate prometheus-k8s admin/controller.controller
# controller SAAS → 'error' after ~5s
# after 12m50s: check_prometheus_targets prometheus-k8s 0 failed after 30 retries
Charm revs observed failing: juju-controller-157 (4.1-beta2 main),
juju-controller-328 (4.0.15.x). 3.6.28 / juju-controller-361 does not
exhibit the failure.
Tracked from juju side at https://warthogs.atlassian.net/browse/JUJU-10266.
Root cause
lib/charms/prometheus_k8s/v0/prometheus_scrape.py:_set_unit_ip calls
unit_ip = str(self._charm.model.get_binding(relation).network.bind_address)
binding.network.bind_address accesses ops.model.NetworkInterface.__init__,
which calls ipaddress.ip_network(address). When the controller pod has just
been created on k8s, network_get returns the service-endpoints DNS FQDN
(<unit>.controller-service-endpoints.<ns>.svc.cluster.local, matching the
ControllerServiceFQDNTemplate in juju/internal/provider/kubernetes/constants/constants.go),
which is not a valid CIDR. The ValueError propagates as an uncaught
exception, the hook fails, and because the charm has no containers: and no
refresh_event, the IP is not re-evaluated — every ~30 s retry repeats
the same race.
Note: the same
_set_unit_ipline exists in the upstreammainat
revision 58 ofprometheus_k8s/v0/prometheus_scrape.py, with the same
missing guard.
Suggested fixes
Either of:
-
Charmlib — wrap the
bind_addressaccess in atry/exceptand defer
the relation-data write via a deferred event handler when the binding is
unresolved; or generally wrapset_scrape_job_specso that_set_unit_ip
failures do not fail the hook. -
Charm — add a
containers:stanza withrefresh_event: <container>-pebble-ready
inmetadata.yamland observe PebbleReady formetrics-endpoint. This
also suppresses the "0 containers in metadata.yaml" warning and gives
the IP a retry path through the operator framework.
Local mitigation until either lands
In juju/tests/suites/controllercharm/prometheus.sh, after juju relate,
add a wait_for "controller SAAS status == active" before
check_prometheus_targets, so failures surface with the actual error rather
than a 12m50s retry timeout (see https://warthogs.atlassian.net/browse/JUJU-10266).
Likely related
- #93 "Wrong host address for metrics scrape job" — same
_set_unit_ip
line (main @ line 1692 / rev-157 @ line 1554) returns a non-IP target
without crashing. I believe #93 and this issue share a root cause:
missing guard onbind_addressand no retry path when the binding is
not yet IP-resolvable. - #112 "metrics-endpoint-relation-created on every unit fail in cluster
mode" — different failure (APIError: user "juju-metrics-r1" already exists) on the same hook, but a third race condition on the relation
lifecycle.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.