metrics-endpoint-relation-created on every unit fail in cluster mode

I have a juju controller 3 node cluster.

I have a alloy-vm charm which can integrate with the controller via its metrics-endpoint

When I relate it to my monitoring charm "alloy-vm" which is capable of scraping metrics the relation hook ends up in an error state.

juju switch admin/controller juju integrate controller:metrics-endpoint alloy-vm:metrics-endpoint

It ends up in this:

File "/var/lib/juju/agents/unit-controller-0/charm/venv/ops/framework.py", line 947, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-controller-0/charm/./src/charm.py", line 81, in _on_metrics_endpoint_relation_created self.control_socket.add_metrics_user(username, password) File "/var/lib/juju/agents/unit-controller-0/charm/src/controlsocket.py", line 133, in add_metrics_user resp = self._request( ^^^^^^^^^^^^^^ File "/var/lib/juju/agents/unit-controller-0/charm/src/controlsocket.py", line 80, in _request response = self._request_raw(method, path, query, headers, data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/juju/agents/unit-controller-0/charm/src/controlsocket.py", line 126, in _request_raw raise APIError(body, code, status, message) controlsocket.APIError: user "juju-metrics-r1" already exists unit-controller-0: 18:43:40 ERROR juju.worker.uniter.operation hook "metrics-endpoint-relation-created" (via hook dispatching script: dispatch) failed: exit status 1

Start with _on_metrics_endpoint_relation_created in src/charm.py and follow add_metrics_user in src/controlsocket.py, especially the APIError raised by _request_raw. Reproduce the metrics-endpoint integration in a three-node controller cluster and verify that repeated relation-created events do not leave the hook in an error state when the juju-metrics-r1 user already exists.