Temporal Is not able to connect to Cassandra even when one node is down in a cluster

Expected Behavior

We have a 3 node cluster Even when one node is down we are expecting the temporal to work by connecting to other two nodes.

Actual Behavior:

We are doing the load test and we observed Temporal is not able to connect to Cassandra even when ONE node is down

Steps to Reproduce the Problem

1. In a cluster make sure one node in Cassandra is down. The temporal pods are not able to connect to remaining other two nodes.

Error:
2022/04/14 23:41:44 error: failed to connect to XX.XXX.XX.7:9042 due to error: write tcp XX.XXX.XX.79:44342->XX.XXX.XX.7:9042: write: connection reset by peer
unable to dial control conn XX.XXX.XX.7:9042 gocql: no response received from cassandra within timeout period

Below is my Configuration:

cassandra:
hosts: [“XX.XXX.XX.7,XX.XXX.XX.9,XX.XXX.XX.10”]
port: 9042
keyspace: temporal
user: “temporal”
password: “XXXXXXX”
existingSecret: “”
replicationFactor: 3 (Tried both 1 and 3)
consistency:
default:
consistency: “local_quorum”
serialConsistency: “local_serial”
tls:
enabled: true
enableHostVerification: false

Note: We are mentioning the cluster info with comma separated ip’s. We did updated Replication factor with 1 ,3 both did not worked.

Specifications

• Version: 1.13
• Platform: Temporal is deployed in Azure . Cassandra is managed by [azure]

Cassandra Configuration:

No source file or test is named. Start by reproducing the failure with the three-node Cassandra configuration, one node down, and the reported consistency settings, then trace how Temporal handles the connection-reset and timeout. Done means Temporal remains usable through the two healthy nodes under the described load test.