temporalio / temporalio/temporal
Membership: emit per-service reachable/available/draining member gauges
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
Is your feature request related to a problem? Please describe.
Investigating #10730 (serviceResolver.Lookup() routing to draining/stopping members during rolling restarts) and #11108 (no operator-facing way to evict a gray-failed host) surfaced a related gap: the membership package currently emits almost no metrics, so operators have no visibility into ring health before it turns into a routing incident like these two.
Confirmed by reading common/membership/: the only existing metric, MembershipChangedCounter, is emitted from outside the package (service/history/shard/ownership.go), not from membership itself. Neither monitor.go nor service_resolver.go accept a metrics.Handler today. There's no gauge for how many hosts are reachable, how many are actually accepting traffic (non-draining), or how many are draining, per service.
Describe the solution you'd like
Add three per-service gauges, emitted on every ring refresh in serviceResolver.refreshLocked():
membership_reachable_members — from the existing MemberCount()
membership_available_members — from the existing AvailableMemberCount() (non-draining)
membership_draining_members — the difference between the two
All tagged with the existing metrics.ServiceNameTag(service) for a per-service (frontend/history/matching/worker) breakdown.
Implementation shape:
Add MetricsHandler metrics.Handler to factoryParams in common/membership/ringpop/factory.go (fx already provides this elsewhere, so it's just a new field)
Thread it through newMonitor → newServiceResolver
Emit the three gauges inside refreshLocked() right after the new ringAndHosts is stored, following the existing metrics.NewGaugeDef(...) + .With(handler).Record(...) pattern used elsewhere (e.g. NumShardsGauge in service/history/shard/controller_impl.go)
This is purely additive — no change to Lookup()/routing behavior — so it doesn't collide with #10730's fix landing in parallel, and gives operators (and dashboards/alerts) the missing signal to catch ring-health degradation before it causes a routing incident.
Describe alternatives you've considered
Waiting for #10730/#11108 to land first — but those solve routing correctness, not observability; this is complementary, not redundant, and can land independently.
Logging only (current state) — the existing "Current reachable members" log line in service_resolver.go isn't queryable/alertable the way a metric is.
Additional context
Happy to contribute a PR for this if the team is open to the approach — I've already mapped the exact call sites and confirmed the existing patterns to follow (gauge defs, service name tagging, refresh-cycle hook point).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in common/membership/ringpop/factory.go and trace factoryParams through newMonitor and newServiceResolver into service_resolver.go. Inspect serviceResolver.refreshLocked(), the existing MemberCount() and AvailableMemberCount() methods, and the NumShardsGauge pattern in service/history/shard/controller_impl.go. Done means each refresh records reachable, available, and draining per-service gauges with metrics.ServiceNameTag(service).
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, observability-sre
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100