canonical / canonical/cos-coordinated-workers

metrics scrape_job for coordinator's nginx is misconfigured when num_units > 1

Open
#94 0 comments 0 reactions 0 assignees View on GitHub
Status: Triage Type: Bug
Dominant language
Python
Stars
1
Forks
2
PR merge metrics
No merged PRs in 30d

Description

### Bug Description

Unconfirmed, but I suspect we only emit nginx metrics from the Coordinator's leader unit when Coordinator is scaled. I see Coordinator instantiates a metrics relation with a scrape job for all workers [and this particular coordinator unit](https://github.com/canonical/cos-coordinated-workers/blob/245d0c23922c38e76590248f5d30574d67d624cb/src/coordinated_workers/coordinator.py#L651-L659). If a coordinator is scaled, all units will independently create this same job (with their own unit's nginx) but **only the leader will send that data because MetricsEndpointProvider only sends data from the leader**. So what I think we'll see is nginx logs from only the leader's unit and no others

### To Reproduce

deploy with coordinator having scale=2 and scrape metrics

### Environment

.

### Relevant log output

```shell
.
```

### Additional context

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/coordinated_workers/coordinator.py at the metrics relation and scrape-job setup around lines 651-659, then inspect how MetricsEndpointProvider handles data from scaled units. Reproduce the issue with Coordinator scaled to two units and scrape the metrics. Done means nginx metrics are emitted for every coordinator unit rather than only the leader.

Written by the indexing model from the issue text.

Assessment

Tech stack
nginx, python
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.