canonical / canonical/cos-coordinated-workers
Non-deterministic gathering of Loki endpoints and Tracing receivers leads to too many unnecessary relation changed events
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
The coordinated workers often have to execute too many unnecessary "relation changed" events due to the coordinator writing unordered data to relation data.
For example, across two hooks, the relation data written by the coordinator changes in the following way
```bash
< loki-backend/0:
---
> oki-backend/0:
12,16c12,16
< charm_tracing_receivers: '{"jaeger_thrift_http": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/charm-tracing/jaeger_thrift_http/",
< "otlp_http": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/charm-tracing/otlp_http/"}'
< loki_endpoints: '{"otel-collector/0": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-0/push",
< "otel-collector/1": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-1/push",
< "otel-collector/2": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-2/push"}'
---
> charm_tracing_receivers: '{"otlp_http": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/charm-tracing/otlp_http/",
> "jaeger_thrift_http": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/charm-tracing/jaeger_thrift_http/"}'
> loki_endpoints: '{"otel-collector/1": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-1/push",
> "otel-collector/2": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-2/push",
> "otel-collector/0": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-0/push"}'
```
The reason seems to be related to non-deterministic operations for worker telemetry purposes. We get the list of addresses needed by looking at units, but `relation.units` in `ops` returns a set, which is non-deterministic by nature.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the telemetry code that gathers units through ops relation.units and writes charm_tracing_receivers and loki_endpoints to relation data. Verify the endpoint mappings are serialized in a stable order, and add or update tests showing repeated hooks do not produce unnecessary relation changed events.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100