temporalio / temporalio/temporal

Ringpop membership churn after upgrade to v1.30.x

Open
#9,987 10 comments 4 reactions 1 assignee View on GitHub

@dnr is already working on this.

Since Apr 30, 2026.

potential-bug
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Expected Behavior

After upgrading from v1.29.x to v1.30.x, the ringpop membership ring should
converge within seconds of pods starting. Each service (frontend, history,
matching, worker) should see only the currently running pods as reachable
members, and inter-service gRPC calls should succeed normally.

Actual Behavior

After upgrading to v1.30.4, the cluster exhibits permanent ringpop membership
churn (~4000 ring change events per 30 seconds). The membership ring sees
17 members when only 4 pods are running — the extra 13 are zombie IPs from
previous pod restarts still present in Cassandra. SWIM continuously pings
these dead IPs, causing them to cycle alive→suspect→faulty indefinitely.
This results in continuous EOF errors between frontend→matching and
frontend→history, making workflow execution impossible. The cluster does
not self-heal.

  Ring churn log (repeating ~140 times/second per pod):
  {"level":"debug","msg":"Received a ring changed event",
   "component":"service-resolver","service":"history",
   "logging-call-at":"common/membership/ringpop/service_resolver.go:252"}

EOF errors between services:

  {"level":"error","msg":"matching client encountered error",
   "service":"frontend",
   "error":"connection error: desc = \"error reading server preface: EOF\"",
   "logging-call-at":"client/matching/metric_client.go:197"}

Ring sees 17 members, only 4 pods running:

  {"level":"info","msg":"Current reachable members",
   "service":"matching",
   "addresses":["10.0.0.154:7235","10.0.0.155:7235", ...17 IPs total...]}

Steps to Reproduce the Problem

1. Run Temporal v1.29.x on Kubernetes with Cassandra backend and perform
   several rolling restarts (simulating normal operations/upgrade process),
   allowing stale pod IPs to accumulate in cluster_membership table
   (TTL = 48h)
2. Upgrade to v1.30.x — commit f7ca7745 ("set page token during ringpop
   bootstrap") is now active, causing fetchCurrentBootstrapHostports() to
   correctly paginate and read ALL rows from cluster_membership instead of
   only the first page as in v1.29.x
3. Observe all pods immediately exhibit ~4000 ring change events per 30
   seconds and inter-service EOF errors upon startup

Specifications

- Version: v1.30.4 (upgraded from v1.29.x)
- Platform: Kubernetes with Cassandra persistence backend
- Introduced by: commit f7ca7745 ("set page token during ringpop bootstrap")
- Root cause: upsertMembershipRecordExpiryDefault = 48h in
  common/membership/ringpop/monitor.go allows stale pod IPs to accumulate
  across restarts; v1.29.x accidentally avoided this via the pagination bug;

  v1.30.x correctly reads all stale rows and injects them into SWIM
- Proposed fix: reduce upsertMembershipRecordExpiryDefault from 48h to ~5m,
  or expose it as a configurable parameter

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.