temporalio / temporalio/temporal

temporal-server "hang" during SIGTERM after loss of Cassandra connection

Open
#1,867 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

potential-bug
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Expected Behavior

When send interrupt signal; temporal-server cleanly shutdown

Actual Behavior

Unable to cleanly shutdown; it "hangs". Normal SIGTERM is ignored; have to SIGKILL.
Only happens after network ; and it does not retry and "hangs"

Steps to Reproduce the Problem

  1. Start temporal-server with config below; no need HostLookup as Cassandra only have one endpoint to talk to (managed Cassandra AstraDB)
  2. Turn off network/wifi (simulate loss of connection to DB)
  3. Run an operation 'tctl wf list'; oobserve failure and reconnect network/wifi
  4. Once network connection is back; confirm normal operation works 'tctl wf list'
  5. Ctrl-C, observe error and "hang", forced to SIGKILL

Cassandra config

    cass-default:
      cassandra:
        hosts: <endpoint>-ap-southeast-1.db.astra.datastax.com
        port: 29042
...
        disableInitialHostLookup: true
        keyspace: "temporal2"
    cass-visibility:
      cassandra:
        hosts: <endpoint>-ap-southeast-1.db.astra.datastax.com
        port: 29042
...
        disableInitialHostLookup: true
        keyspace: "temporal_visibility2"

Observation

It looks as though after hitting "read: connection reset by peer", the node is marked unhealthy. When there is the config disableInitialHostLookup=true it should just retry the known connection after a short while.

Specifications

  • Version: temporal version 1.12.0
  • Platform: OSX
  • DB: Managed Cassandra AstraDB

Logs

{"level":"error","ts":"2021-08-28T14:42:02.680+0800","msg":"Membership upsert failed.","service":"history","error":"operation UpsertClusterMembership encounter read tcp 192.168.0.104:50393->13.213.36.186:29042: read: connection reset by peer","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
{"level":"error","ts":"2021-08-28T14:42:02.680+0800","msg":"Membership upsert failed.","service":"matching","error":"operation UpsertClusterMembership encounter read tcp 192.168.0.104:50414->13.212.129.122:29042: read: connection reset by peer","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
{"level":"error","ts":"2021-08-28T14:42:02.874+0800","msg":"Operation failed with internal error.","service":"matching","error":"read tcp 192.168.0.104:50410->13.213.36.186:29042: read: connection reset by peer","metric-scope":52,"logging-call-at":"persistenceMetricClients.go:896","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).updateErrorMetric\n\t/Users/leow/GOMOD/temporal/common/persistence/persistenceMetricClients.go:896\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).GetMetadata\n\t/Users/leow/GOMOD/temporal/common/persistence/persistenceMetricClients.go:874\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshNamespacesLocked\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:435\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshNamespaces\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:425\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:409"}
{"level":"error","ts":"2021-08-28T14:42:02.875+0800","msg":"Error refreshing namespace cache","service":"matching","error":"read tcp 192.168.0.104:50410->13.213.36.186:29042: read: connection reset by peer","logging-call-at":"namespaceCache.go:414","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:414"}
{"level":"error","ts":"2021-08-28T14:42:03.937+0800","msg":"Operation failed with internal error.","service":"history","error":"operation UpdateShard encounter read tcp 192.168.0.104:50395->13.213.36.186:29042: read: connection reset by peer","metric-scope":3,"logging-call-at":"persistenceMetricClients.go:194","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/persistence.(*shardPersistenceClient).updateErrorMetric\n\t/Users/leow/GOMOD/temporal/common/persistence/persistenceMetricClients.go:194\ngo.temporal.io/server/common/persistence.(*shardPersistenceClient).UpdateShard\n\t/Users/leow/GOMOD/temporal/common/persistence/persistenceMetricClients.go:176\ngo.temporal.io/server/service/history/shard.(*ContextImpl).updateShardInfoLocked\n\t/Users/leow/GOMOD/temporal/service/history/shard/context_impl.go:778\ngo.temporal.io/server/service/history/shard.(*ContextImpl).UpdateTimerClusterAckLevel\n\t/Users/leow/GOMOD/temporal/service/history/shard/context_impl.go:309\ngo.temporal.io/server/service/history.newTimerQueueActiveProcessor.func2\n\t/Users/leow/GOMOD/temporal/service/history/timerQueueActiveProcessor.go:70\ngo.temporal.io/server/service/history.(*timerQueueAckMgrImpl).updateAckLevel\n\t/Users/leow/GOMOD/temporal/service/history/timerQueueAckMgr.go:401\ngo.temporal.io/server/service/history.(*timerQueueProcessorBase).internalProcessor\n\t/Users/leow/GOMOD/temporal/service/history/timerQueueProcessorBase.go:320\ngo.temporal.io/server/service/history.(*timerQueueProcessorBase).processorPump\n\t/Users/leow/GOMOD/temporal/service/history/timerQueueProcessorBase.go:195"}
{"level":"error","ts":"2021-08-28T14:42:03.938+0800","msg":"Error updating timer ack level for shard","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"timer-queue-processor","cluster-name":"active","error":"operation UpdateShard encounter read tcp 192.168.0.104:50395->13.213.36.186:29042: read: connection reset by peer","logging-call-at":"timerQueueAckMgr.go:403","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/history.(*timerQueueAckMgrImpl).updateAckLevel\n\t/Users/leow/GOMOD/temporal/service/history/timerQueueAckMgr.go:403\ngo.temporal.io/server/service/history.(*timerQueueProcessorBase).internalProcessor\n\t/Users/leow/GOMOD/temporal/service/history/timerQueueProcessorBase.go:320\ngo.temporal.io/server/service/history.(*timerQueueProcessorBase).processorPump\n\t/Users/leow/GOMOD/temporal/service/history/timerQueueProcessorBase.go:195"}
{"level":"error","ts":"2021-08-28T14:42:04.019+0800","msg":"Operation failed with internal error.","service":"worker","error":"read tcp 192.168.0.104:50424->13.213.36.186:29042: read: connection reset by peer","metric-scope":52,"logging-call-at":"persistenceMetricClients.go:896","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).updateErrorMetric\n\t/Users/leow/GOMOD/temporal/common/persistence/persistenceMetricClients.go:896\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).GetMetadata\n\t/Users/leow/GOMOD/temporal/common/persistence/persistenceMetricClients.go:874\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshNamespacesLocked\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:435\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshNamespaces\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:425\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:409"}
{"level":"error","ts":"2021-08-28T14:42:04.019+0800","msg":"Error refreshing namespace cache","service":"worker","error":"read tcp 192.168.0.104:50424->13.213.36.186:29042: read: connection reset by peer","logging-call-at":"namespaceCache.go:414","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:414"}
{"level":"error","ts":"2021-08-28T14:42:04.051+0800","msg":"Operation failed with internal error.","service":"history","error":"operation UpdateShard encounter read tcp 192.168.0.104:50392->13.213.36.186:29042: read: connection reset by peer","metric-scope":3,"logging-call-at":"persistenceMetricClients.go:194","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/persistence.(*shardPersistenceClient).updateErrorMetric\n\t/Users/leow/GOMOD/temporal/common/persistence/persistenceMetricClients.go:194\ngo.temporal.io/server/common/persistence.(*shardPersistenceClient).UpdateShard\n\t/Users/leow/GOMOD/temporal/common/persistence/persistenceMetricClients.go:176\ngo.temporal.io/server/service/history/shard.(*ContextImpl).updateShardInfoLocked\n\t/Users/leow/GOMOD/temporal/service/history/shard/context_impl.go:778\ngo.temporal.io/server/service/history/shard.(*ContextImpl).UpdateTransferClusterAckLevel\n\t/Users/leow/GOMOD/temporal/service/history/shard/context_impl.go:182\ngo.temporal.io/server/service/history.newTransferQueueActiveProcessor.func3\n\t/Users/leow/GOMOD/temporal/service/history/transferQueueActiveProcessor.go:99\ngo.temporal.io/server/service/history.(*transferQueueProcessorBase).updateAckLevel\n\t/Users/leow/GOMOD/temporal/service/history/transferQueueProcessorBase.go:99\ngo.temporal.io/server/service/history.(*queueAckMgrImpl).updateQueueAckLevel\n\t/Users/leow/GOMOD/temporal/service/history/queueAckMgr.go:222\ngo.temporal.io/server/service/history.(*queueProcessorBase).processorPump\n\t/Users/leow/GOMOD/temporal/service/history/queueProcessor.go:241"}
{"level":"error","ts":"2021-08-28T14:42:04.174+0800","msg":"Error updating ack level for shard","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"transfer-queue-processor","cluster-name":"active","error":"operation UpdateShard encounter read tcp 192.168.0.104:50392->13.213.36.186:29042: read: connection reset by peer","operation-result":"OperationFailed","logging-call-at":"queueAckMgr.go:224","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/history.(*queueAckMgrImpl).updateQueueAckLevel\n\t/Users/leow/GOMOD/temporal/service/history/queueAckMgr.go:224\ngo.temporal.io/server/service/history.(*queueProcessorBase).processorPump\n\t/Users/leow/GOMOD/temporal/service/history/queueProcessor.go:241"}
{"level":"error","ts":"2021-08-28T14:42:05.244+0800","msg":"Operation failed with internal error.","service":"worker","error":"read tcp 192.168.0.104:50419->13.212.129.122:29042: read: connection reset by peer","metric-scope":52,"logging-call-at":"persistenceMetricClients.go:896","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).updateErrorMetric\n\t/Users/leow/GOMOD/temporal/common/persistence/persistenceMetricClients.go:896\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).GetMetadata\n\t/Users/leow/GOMOD/temporal/common/persistence/persistenceMetricClients.go:874\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshNamespacesLocked\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:435\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshNamespaces\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:425\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:409"}
{"level":"error","ts":"2021-08-28T14:42:05.265+0800","msg":"Error refreshing namespace cache","service":"worker","error":"read tcp 192.168.0.104:50419->13.212.129.122:29042: read: connection reset by peer","logging-call-at":"namespaceCache.go:414","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/Users/leow/GOMOD/temporal/common/cache/namespaceCache.go:414"}



{"level":"info","ts":"2021-08-28T14:42:21.560+0800","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-add-search-attributes-task-queue/2","wf-task-queue-type":"Activity","lifecycle":"Stopped","logging-call-at":"taskQueueManager.go:259"}
{"level":"info","ts":"2021-08-28T14:43:05.635+0800","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Starting","wf-task-queue-name":"/_sys/temporal-sys-add-search-attributes-task-queue/2","wf-task-queue-type":"Activity","logging-call-at":"matchingEngine.go:206"}
{"level":"info","ts":"2021-08-28T14:43:05.635+0800","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Started","wf-task-queue-name":"/_sys/temporal-sys-add-search-attributes-task-queue/2","wf-task-queue-type":"Activity","logging-call-at":"matchingEngine.go:209"}
^C{"level":"info","ts":"2021-08-28T14:43:51.239+0800","msg":"Received interrupt signal, stopping the server.","value":"interrupt","logging-call-at":"server.go:230"}
{"level":"info","ts":"2021-08-28T14:43:51.240+0800","msg":"Get dynamic config","name":"frontend.shutdownDrainDuration","value":0,"default-value":0,"logging-call-at":"config.go:79"}
{"level":"info","ts":"2021-08-28T14:43:51.240+0800","msg":"ShutdownHandler: Updating rpc health status to ShuttingDown","service":"frontend","logging-call-at":"service.go:402"}
{"level":"info","ts":"2021-08-28T14:43:51.240+0800","msg":"ShutdownHandler: Waiting for others to discover I am unhealthy","service":"frontend","logging-call-at":"service.go:405"}
{"level":"info","ts":"2021-08-28T14:43:51.240+0800","msg":"ShutdownHandler: Draining traffic","service":"frontend","logging-call-at":"service.go:411"}
{"level":"info","ts":"2021-08-28T14:43:51.240+0800","msg":"Get dynamic config","name":"history.shutdownDrainDuration","value":0,"default-value":0,"logging-call-at":"config.go:79"}
{"level":"info","ts":"2021-08-28T14:43:51.241+0800","msg":"ShutdownHandler: Evicting self from membership ring","service":"history","logging-call-at":"service.go:226"}
{"level":"info","ts":"2021-08-28T14:43:51.240+0800","msg":"RuntimeMetricsReporter stopped","service":"worker","logging-call-at":"runtime.go:160"}
{"level":"info","ts":"2021-08-28T14:43:51.240+0800","msg":"ShutdownHandler: Evicting self from membership ring","service":"matching","logging-call-at":"service.go:151"}
{"level":"info","ts":"2021-08-28T14:43:51.241+0800","msg":"RuntimeMetricsReporter stopped","service":"frontend","logging-call-at":"runtime.go:160"}
{"level":"info","ts":"2021-08-28T14:43:51.242+0800","msg":"Received a ring changed event","service":"history","component":"service-resolver","service":"frontend","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.242+0800","msg":"Received a ring changed event","service":"history","component":"service-resolver","service":"matching","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.245+0800","msg":"Current reachable members","service":"history","component":"service-resolver","service":"matching","addresses":[],"logging-call-at":"rpServiceResolver.go:266"}
{"level":"info","ts":"2021-08-28T14:43:51.243+0800","msg":"frontend stopped","service":"frontend","logging-call-at":"service.go:426"}
{"level":"info","ts":"2021-08-28T14:43:51.243+0800","msg":"Received a ring changed event","service":"matching","component":"service-resolver","service":"frontend","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.243+0800","msg":"Received a ring changed event","service":"matching","component":"service-resolver","service":"matching","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.245+0800","msg":"Current reachable members","service":"matching","component":"service-resolver","service":"matching","addresses":[],"logging-call-at":"rpServiceResolver.go:266"}
{"level":"info","ts":"2021-08-28T14:43:51.246+0800","msg":"ShutdownHandler: Waiting for others to discover I am unhealthy","service":"history","logging-call-at":"service.go:229"}
{"level":"info","ts":"2021-08-28T14:43:51.243+0800","msg":"Received a ring changed event","service":"matching","component":"service-resolver","service":"worker","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.247+0800","msg":"ShutdownHandler: Initiating shardController shutdown","service":"history","logging-call-at":"service.go:232"}
{"level":"info","ts":"2021-08-28T14:43:51.244+0800","msg":"Received a ring changed event","service":"history","component":"service-resolver","service":"frontend","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.247+0800","msg":"none","service":"history","component":"shard-controller","address":"127.0.0.1:7234","lifecycle":"Stopping","logging-call-at":"controller_impl.go:383"}
{"level":"info","ts":"2021-08-28T14:43:51.247+0800","msg":"none","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","lifecycle":"Stopping","component":"shard-engine","logging-call-at":"controller_impl.go:460"}
{"level":"info","ts":"2021-08-28T14:43:51.244+0800","msg":"Received a ring changed event","service":"history","component":"service-resolver","service":"matching","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.244+0800","msg":"Received a ring changed event","service":"history","component":"service-resolver","service":"history","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.244+0800","msg":"Received a ring changed event","service":"history","component":"service-resolver","service":"worker","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.244+0800","msg":"ShutdownHandler: Waiting for others to discover I am unhealthy","service":"matching","logging-call-at":"service.go:153"}
{"level":"info","ts":"2021-08-28T14:43:51.245+0800","msg":"Received a ring changed event","service":"history","component":"service-resolver","service":"history","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.245+0800","msg":"Received a ring changed event","service":"history","component":"service-resolver","service":"worker","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.246+0800","msg":"Received a ring changed event","service":"matching","component":"service-resolver","service":"frontend","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.246+0800","msg":"Received a ring changed event","service":"matching","component":"service-resolver","service":"matching","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.246+0800","msg":"Received a ring changed event","service":"matching","component":"service-resolver","service":"history","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.246+0800","msg":"Received a ring changed event","service":"matching","component":"service-resolver","service":"worker","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.243+0800","msg":"Received a ring changed event","service":"matching","component":"service-resolver","service":"history","logging-call-at":"rpServiceResolver.go:219"}
{"level":"info","ts":"2021-08-28T14:43:51.248+0800","msg":"Current reachable members","service":"history","component":"service-resolver","service":"history","addresses":[],"logging-call-at":"rpServiceResolver.go:266"}
{"level":"info","ts":"2021-08-28T14:43:51.248+0800","msg":"none","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"history-engine","lifecycle":"Stopping","logging-call-at":"historyEngine.go:309"}
{"level":"info","ts":"2021-08-28T14:43:51.248+0800","msg":"none","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"transfer-queue-processor","cluster-name":"active","lifecycle":"Stopping","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:166"}
{"level":"info","ts":"2021-08-28T14:43:51.248+0800","msg":"Current reachable members","service":"matching","component":"service-resolver","service":"history","addresses":[],"logging-call-at":"rpServiceResolver.go:266"}
{"level":"info","ts":"2021-08-28T14:43:51.260+0800","msg":"Queue processor pump shut down.","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"transfer-queue-processor","cluster-name":"active","logging-call-at":"queueProcessor.go:249"}
{"level":"info","ts":"2021-08-28T14:43:51.260+0800","msg":"Task processor shutdown.","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"transfer-queue-processor","cluster-name":"active","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.260+0800","msg":"none","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"transfer-queue-processor","cluster-name":"active","lifecycle":"Stopped","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:179"}
{"level":"info","ts":"2021-08-28T14:43:51.260+0800","msg":"Timer queue processor pump shutting down.","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:202"}
{"level":"info","ts":"2021-08-28T14:43:51.260+0800","msg":"Timer processor exiting.","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:203"}
{"level":"info","ts":"2021-08-28T14:43:51.260+0800","msg":"Task processor shutdown.","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.260+0800","msg":"Timer queue processor stopped.","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:183"}
{"level":"info","ts":"2021-08-28T14:43:51.248+0800","msg":"Get dynamic config","name":"matching.shutdownDrainDuration","value":0,"default-value":0,"logging-call-at":"config.go:79"}
{"level":"info","ts":"2021-08-28T14:43:51.261+0800","msg":"none","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"visibility-queue-processor","lifecycle":"Stopping","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:166"}
{"level":"info","ts":"2021-08-28T14:43:51.261+0800","msg":"RuntimeMetricsReporter stopped","service":"matching","logging-call-at":"runtime.go:160"}
{"level":"info","ts":"2021-08-28T14:43:51.261+0800","msg":"Queue processor pump shut down.","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"visibility-queue-processor","logging-call-at":"queueProcessor.go:249"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"Task processor shutdown.","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"visibility-queue-processor","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"visibility-queue-processor","lifecycle":"Stopped","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:179"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","component":"history-engine","lifecycle":"Stopped","logging-call-at":"historyEngine.go:328"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":3,"address":"127.0.0.1:7234","shard-item":"0xc000135c80","lifecycle":"Stopped","component":"shard-engine","logging-call-at":"controller_impl.go:463"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","lifecycle":"Stopping","component":"shard-engine","logging-call-at":"controller_impl.go:460"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"history-engine","lifecycle":"Stopping","logging-call-at":"historyEngine.go:309"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"transfer-queue-processor","cluster-name":"active","lifecycle":"Stopping","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:166"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"Queue processor pump shut down.","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"transfer-queue-processor","cluster-name":"active","logging-call-at":"queueProcessor.go:249"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"Task processor shutdown.","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"transfer-queue-processor","cluster-name":"active","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"transfer-queue-processor","cluster-name":"active","lifecycle":"Stopped","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:179"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"Timer queue processor pump shutting down.","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:202"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"Timer processor exiting.","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:203"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"Task processor shutdown.","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"Timer queue processor stopped.","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:183"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"visibility-queue-processor","lifecycle":"Stopping","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:166"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"Queue processor pump shut down.","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"visibility-queue-processor","logging-call-at":"queueProcessor.go:249"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"Task processor shutdown.","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"visibility-queue-processor","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"visibility-queue-processor","lifecycle":"Stopped","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:179"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","component":"history-engine","lifecycle":"Stopped","logging-call-at":"historyEngine.go:328"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":4,"address":"127.0.0.1:7234","shard-item":"0xc000854000","lifecycle":"Stopped","component":"shard-engine","logging-call-at":"controller_impl.go:463"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","lifecycle":"Stopping","component":"shard-engine","logging-call-at":"controller_impl.go:460"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"history-engine","lifecycle":"Stopping","logging-call-at":"historyEngine.go:309"}
{"level":"info","ts":"2021-08-28T14:43:51.283+0800","msg":"none","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"transfer-queue-processor","cluster-name":"active","lifecycle":"Stopping","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:166"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Queue processor pump shut down.","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"transfer-queue-processor","cluster-name":"active","logging-call-at":"queueProcessor.go:249"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Task processor shutdown.","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"transfer-queue-processor","cluster-name":"active","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"none","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"transfer-queue-processor","cluster-name":"active","lifecycle":"Stopped","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:179"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Timer queue processor pump shutting down.","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:202"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Timer processor exiting.","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:203"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Task processor shutdown.","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Timer queue processor stopped.","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:183"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"none","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"visibility-queue-processor","lifecycle":"Stopping","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:166"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Queue processor pump shut down.","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"visibility-queue-processor","logging-call-at":"queueProcessor.go:249"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Task processor shutdown.","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"visibility-queue-processor","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"none","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"visibility-queue-processor","lifecycle":"Stopped","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:179"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"none","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","component":"history-engine","lifecycle":"Stopped","logging-call-at":"historyEngine.go:328"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"none","service":"history","shard-id":1,"address":"127.0.0.1:7234","shard-item":"0xc000135800","lifecycle":"Stopped","component":"shard-engine","logging-call-at":"controller_impl.go:463"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"none","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","lifecycle":"Stopping","component":"shard-engine","logging-call-at":"controller_impl.go:460"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"none","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"history-engine","lifecycle":"Stopping","logging-call-at":"historyEngine.go:309"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"none","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"transfer-queue-processor","cluster-name":"active","lifecycle":"Stopping","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:166"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Queue processor pump shut down.","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"transfer-queue-processor","cluster-name":"active","logging-call-at":"queueProcessor.go:249"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Task processor shutdown.","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"transfer-queue-processor","cluster-name":"active","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"none","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"transfer-queue-processor","cluster-name":"active","lifecycle":"Stopped","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:179"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Timer queue processor pump shutting down.","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:202"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Timer processor exiting.","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:203"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Task processor shutdown.","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Timer queue processor stopped.","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"timer-queue-processor","cluster-name":"active","component":"timer-queue-processor","logging-call-at":"timerQueueProcessorBase.go:183"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"none","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"visibility-queue-processor","lifecycle":"Stopping","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:166"}
{"level":"info","ts":"2021-08-28T14:43:51.284+0800","msg":"Queue processor pump shut down.","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"visibility-queue-processor","logging-call-at":"queueProcessor.go:249"}
{"level":"info","ts":"2021-08-28T14:43:51.285+0800","msg":"Task processor shutdown.","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"visibility-queue-processor","logging-call-at":"taskProcessor.go:155"}
{"level":"info","ts":"2021-08-28T14:43:51.285+0800","msg":"none","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"visibility-queue-processor","lifecycle":"Stopped","component":"transfer-queue-processor","logging-call-at":"queueProcessor.go:179"}
{"level":"info","ts":"2021-08-28T14:43:51.285+0800","msg":"none","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","component":"history-engine","lifecycle":"Stopped","logging-call-at":"historyEngine.go:328"}
{"level":"info","ts":"2021-08-28T14:43:51.285+0800","msg":"none","service":"history","shard-id":2,"address":"127.0.0.1:7234","shard-item":"0xc000135a00","lifecycle":"Stopped","component":"shard-engine","logging-call-at":"controller_impl.go:463"}
{"level":"info","ts":"2021-08-28T14:43:51.285+0800","msg":"none","service":"history","component":"shard-controller","address":"127.0.0.1:7234","lifecycle":"Stopped","logging-call-at":"controller_impl.go:178"}
{"level":"info","ts":"2021-08-28T14:43:51.285+0800","msg":"ShutdownHandler: Waiting for traffic to drain","service":"history","logging-call-at":"service.go:234"}
{"level":"info","ts":"2021-08-28T14:43:51.285+0800","msg":"ShutdownHandler: No longer taking rpc requests","service":"history","logging-call-at":"service.go:237"}
{"level":"info","ts":"2021-08-28T14:43:51.285+0800","msg":"Replication task fetchers stopped.","service":"history","logging-call-at":"replicationTaskFetcher.go:171"}
{"level":"info","ts":"2021-08-28T14:43:51.285+0800","msg":"RuntimeMetricsReporter stopped","service":"history","logging-call-at":"runtime.go:160"}
{"level":"info","ts":"2021-08-28T14:43:51.285+0800","msg":"history stopped","service":"history","logging-call-at":"service.go:246"}
{"level":"error","ts":"2021-08-28T14:43:54.988+0800","msg":"Membership upsert failed.","service":"history","error":"operation UpsertClusterMembership encounter session has been closed","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
{"level":"error","ts":"2021-08-28T14:43:55.228+0800","msg":"Membership upsert failed.","service":"frontend","error":"operation UpsertClusterMembership encounter session has been closed","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
^C




{"level":"error","ts":"2021-08-28T14:44:08.989+0800","msg":"Membership upsert failed.","service":"history","error":"operation UpsertClusterMembership encounter session has been closed","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
{"level":"error","ts":"2021-08-28T14:44:10.228+0800","msg":"Membership upsert failed.","service":"frontend","error":"operation UpsertClusterMembership encounter session has been closed","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
{"level":"warn","ts":"2021-08-28T14:44:20.663+0800","msg":"Failed to cancel outstanding poller.","service":"frontend","wf-task-queue-name":"/_sys/temporal-sys-history-scanner-taskqueue-0/1","error":"Not enough hosts to serve the request","logging-call-at":"workflowHandler.go:3399"}
{"level":"warn","ts":"2021-08-28T14:44:20.747+0800","msg":"Failed to cancel outstanding poller.","service":"frontend","wf-task-queue-name":"Mojaves-iMac.local:dae73def-31d3-4e33-8ea2-f3e4bb952e32","error":"Not enough hosts to serve the request","logging-call-at":"workflowHandler.go:3399"}
{"level":"error","ts":"2021-08-28T14:44:20.990+0800","msg":"Membership upsert failed.","service":"history","error":"operation UpsertClusterMembership encounter session has been closed","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
{"level":"warn","ts":"2021-08-28T14:44:21.156+0800","msg":"Failed to cancel outstanding poller.","service":"frontend","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","error":"Not enough hosts to serve the request","logging-call-at":"workflowHandler.go:3399"}
{"level":"error","ts":"2021-08-28T14:44:24.229+0800","msg":"Membership upsert failed.","service":"frontend","error":"operation UpsertClusterMembership encounter session has been closed","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
^C^C^C^C^C{"level":"error","ts":"2021-08-28T14:44:30.991+0800","msg":"Membership upsert failed.","service":"history","error":"operation UpsertClusterMembership encounter session has been closed","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
{"level":"error","ts":"2021-08-28T14:44:36.235+0800","msg":"Membership upsert failed.","service":"frontend","error":"operation UpsertClusterMembership encounter session has been closed","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
{"level":"error","ts":"2021-08-28T14:44:42.993+0800","msg":"Membership upsert failed.","service":"history","error":"operation UpsertClusterMembership encounter session has been closed","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
{"level":"error","ts":"2021-08-28T14:44:50.237+0800","msg":"Membership upsert failed.","service":"frontend","error":"operation UpsertClusterMembership encounter session has been closed","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}

{"level":"error","ts":"2021-08-28T14:44:52.995+0800","msg":"Membership upsert failed.","service":"history","error":"operation UpsertClusterMembership encounter session has been closed","logging-call-at":"rpMonitor.go:276","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/Users/leow/GOMOD/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/Users/leow/GOMOD/temporal/common/membership/rpMonitor.go:276"}
''make: *** [server] Killed: 9

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure by starting temporal-server, running tctl wf list, interrupting network access to Cassandra, restoring it, and sending Ctrl-C. Start by reading common/membership/rpMonitor.go, common/cache/namespaceCache.go, and common/persistence/persistenceMetricClients.go, then trace the shutdown path; done means SIGTERM exits cleanly after the connection-loss scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, go
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.