temporalio / temporalio/temporal

Membership: emit per-service reachable/available/draining member gauges

Open
#11,146 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Is your feature request related to a problem? Please describe.

Investigating #10730 (serviceResolver.Lookup() routing to draining/stopping members during rolling restarts) and #11108 (no operator-facing way to evict a gray-failed host) surfaced a related gap: the membership package currently emits almost no metrics, so operators have no visibility into ring health before it turns into a routing incident like these two.

Confirmed by reading common/membership/: the only existing metric, MembershipChangedCounter, is emitted from outside the package (service/history/shard/ownership.go), not from membership itself. Neither monitor.go nor service_resolver.go accept a metrics.Handler today. There's no gauge for how many hosts are reachable, how many are actually accepting traffic (non-draining), or how many are draining, per service.

Describe the solution you'd like

Add three per-service gauges, emitted on every ring refresh in serviceResolver.refreshLocked():

membership_reachable_members — from the existing MemberCount()
membership_available_members — from the existing AvailableMemberCount() (non-draining)
membership_draining_members — the difference between the two

All tagged with the existing metrics.ServiceNameTag(service) for a per-service (frontend/history/matching/worker) breakdown.

Implementation shape:

Add MetricsHandler metrics.Handler to factoryParams in common/membership/ringpop/factory.go (fx already provides this elsewhere, so it's just a new field)
Thread it through newMonitor → newServiceResolver
Emit the three gauges inside refreshLocked() right after the new ringAndHosts is stored, following the existing metrics.NewGaugeDef(...) + .With(handler).Record(...) pattern used elsewhere (e.g. NumShardsGauge in service/history/shard/controller_impl.go)

This is purely additive — no change to Lookup()/routing behavior — so it doesn't collide with #10730's fix landing in parallel, and gives operators (and dashboards/alerts) the missing signal to catch ring-health degradation before it causes a routing incident.

Describe alternatives you've considered

Waiting for #10730/#11108 to land first — but those solve routing correctness, not observability; this is complementary, not redundant, and can land independently.
Logging only (current state) — the existing "Current reachable members" log line in service_resolver.go isn't queryable/alertable the way a metric is.

Additional context

Happy to contribute a PR for this if the team is open to the approach — I've already mapped the exact call sites and confirmed the existing patterns to follow (gauge defs, service name tagging, refresh-cycle hook point).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in common/membership/ringpop/factory.go and trace factoryParams through newMonitor and newServiceResolver into service_resolver.go. Inspect serviceResolver.refreshLocked(), the existing MemberCount() and AvailableMemberCount() methods, and the NumShardsGauge pattern in service/history/shard/controller_impl.go. Done means each refresh records reachable, available, and draining per-service gauges with metrics.ServiceNameTag(service).

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, observability-sre
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.