canonical / canonical/cos-coordinated-workers

All workers are simultaneously restarted if any worker is scaled or reconfigured

Open
#39 1 comment 0 reactions 0 assignees View on GitHub
Status: Triage Type: Bug
Dominant language
Python
Stars
1
Forks
2
PR merge metrics
No merged PRs in 30d

Description

### Bug Description

All workers are provided the same configuration file, like this for tempo:

`juju show-unit tempo-worker-compactor/0 | yq -e '.tempo-worker-compactor/0.relation-info[0].application-data.worker_config' | yq` (with some sections truncated)

```yaml
auth_enabled: false
compactor:
...
ring:
kvstore:
store: memberlist
distributor:
receivers:
...
ring:
kvstore:
store: memberlist
ingester:
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 1
...
memberlist:
abort_if_cluster_join_fails: false
bind_port: 7946
join_members:
- tempo-worker-compactor-0.tempo-worker-compactor-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-distributor-0.tempo-worker-distributor-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-ingester-0.tempo-worker-ingester-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-ingester-1.tempo-worker-ingester-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-metrics-generator-0.tempo-worker-metrics-generator-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-querier-0.tempo-worker-querier-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-query-frontend-0.tempo-worker-query-frontend-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
tls_enabled: false
metrics_generator:
ring:
kvstore:
store: memberlist
storage:
...
overrides:
defaults:
metrics_generator:
processors:
- span-metrics
- service-graphs
querier:
frontend_worker:
frontend_address: tempo-worker-query-frontend-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:9096
server:
grpc_listen_port: 9096
http_listen_port: 3200
storage:
...
```

When this configuration is updated, it is pushed to all workers at once which triggers a relation-changed event on each worker at the same time. These workers then restart their workloads to update their configuration. I'm unclear whether this is a graceful restart or a hard one - work in flight may be lost when this restart occurs.

A case where this happens is when users `juju add-unit`. The above configuration includes an address for all workers to establish a gossip ring (see `memberlist.join_members` above). When a unit of **any** worker is added, it is added to this list which triggers a configuration change to **all** workers. This then triggers a simultaneously restart for all workers to adopt the new configuration. This means that if a user wants to increase the scale of the ingestor, say due to high load, they'll induce a temporary halt of all work (maybe with losing some in-flight work due to the restarts? not sure).

Since workers are all sent the full worker configuration (including config for roles which they aren't), this means that if any worker configuration is updated then all workers will be restarted. This is not such a problem atm though, as I don't see any user-facing configuration that can trigger this

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the relation-changed handling that applies the shared worker configuration and the workload restart path. Reproduce the behavior with juju add-unit and observe which workers restart; done means a configuration change for one worker no longer unnecessarily restarts every worker, with the restart behavior and any in-flight work impact understood.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.