canonical / canonical/cos-coordinated-workers

Non-deterministic gathering of Loki endpoints and Tracing receivers leads to too many unnecessary relation changed events

Open
#166 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
2
PR merge metrics
No merged PRs in 30d

Description

The coordinated workers often have to execute too many unnecessary "relation changed" events due to the coordinator writing unordered data to relation data.

For example, across two hooks, the relation data written by the coordinator changes in the following way
```bash
< loki-backend/0:
---
> oki-backend/0:
12,16c12,16
< charm_tracing_receivers: '{"jaeger_thrift_http": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/charm-tracing/jaeger_thrift_http/",
< "otlp_http": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/charm-tracing/otlp_http/"}'
< loki_endpoints: '{"otel-collector/0": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-0/push",
< "otel-collector/1": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-1/push",
< "otel-collector/2": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-2/push"}'
---
> charm_tracing_receivers: '{"otlp_http": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/charm-tracing/otlp_http/",
> "jaeger_thrift_http": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/charm-tracing/jaeger_thrift_http/"}'
> loki_endpoints: '{"otel-collector/1": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-1/push",
> "otel-collector/2": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-2/push",
> "otel-collector/0": "http://loki.k8s-test-o11y-cos.svc.cluster.local:8080/proxy/loki/otel-collector-0/push"}'

```

The reason seems to be related to non-deterministic operations for worker telemetry purposes. We get the list of addresses needed by looking at units, but `relation.units` in `ops` returns a set, which is non-deterministic by nature.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the telemetry code that gathers units through ops relation.units and writes charm_tracing_receivers and loki_endpoints to relation data. Verify the endpoint mappings are serialized in a stable order, and add or update tests showing repeated hooks do not produce unnecessary relation changed events.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.