canonical / canonical/cos-coordinated-workers
All workers are simultaneously restarted if any worker is scaled or reconfigured
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
### Bug Description
All workers are provided the same configuration file, like this for tempo:
`juju show-unit tempo-worker-compactor/0 | yq -e '.tempo-worker-compactor/0.relation-info[0].application-data.worker_config' | yq` (with some sections truncated)
```yaml
auth_enabled: false
compactor:
...
ring:
kvstore:
store: memberlist
distributor:
receivers:
...
ring:
kvstore:
store: memberlist
ingester:
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 1
...
memberlist:
abort_if_cluster_join_fails: false
bind_port: 7946
join_members:
- tempo-worker-compactor-0.tempo-worker-compactor-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-distributor-0.tempo-worker-distributor-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-ingester-0.tempo-worker-ingester-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-ingester-1.tempo-worker-ingester-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-metrics-generator-0.tempo-worker-metrics-generator-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-querier-0.tempo-worker-querier-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
- tempo-worker-query-frontend-0.tempo-worker-query-frontend-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:7946
tls_enabled: false
metrics_generator:
ring:
kvstore:
store: memberlist
storage:
...
overrides:
defaults:
metrics_generator:
processors:
- span-metrics
- service-graphs
querier:
frontend_worker:
frontend_address: tempo-worker-query-frontend-endpoints.test-self-monitoring-distributed-dfbb44f8.svc.cluster.local:9096
server:
grpc_listen_port: 9096
http_listen_port: 3200
storage:
...
```
When this configuration is updated, it is pushed to all workers at once which triggers a relation-changed event on each worker at the same time. These workers then restart their workloads to update their configuration. I'm unclear whether this is a graceful restart or a hard one - work in flight may be lost when this restart occurs.
A case where this happens is when users `juju add-unit`. The above configuration includes an address for all workers to establish a gossip ring (see `memberlist.join_members` above). When a unit of **any** worker is added, it is added to this list which triggers a configuration change to **all** workers. This then triggers a simultaneously restart for all workers to adopt the new configuration. This means that if a user wants to increase the scale of the ingestor, say due to high load, they'll induce a temporary halt of all work (maybe with losing some in-flight work due to the restarts? not sure).
Since workers are all sent the full worker configuration (including config for roles which they aren't), this means that if any worker configuration is updated then all workers will be restarted. This is not such a problem atm though, as I don't see any user-facing configuration that can trigger this
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the relation-changed handling that applies the shared worker configuration and the workload restart path. Reproduce the behavior with juju add-unit and observe which workers restart; done means a configuration change for one worker no longer unnecessarily restarts every worker, with the restart behavior and any in-flight work impact understood.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100