apache / apache/cassandra-gocql-driver

Rolling restart will not attempt reconnect to recycled nodes (ip's are changing on recycled nodes, kubernetes pods)

Open
#1,678 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.7k
Forks
658
PR merge metrics
No merged PRs in 30d

Description

Please answer these questions before submitting your issue. Thanks!

### What version of Cassandra are you using?
DataStax Astra - 6.8 DSE

### What version of Gocql are you using?
1.20

### What version of Go are you using?

### What did you do?
When rolling restart of the cluster, the application is failing to reconnect.

### What did you expect to see?
Reconnect to the nodes that have been recycled and are now up.

### What did you see instead?
Here's the log when the connection is failing:

```
01:58:48.338 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable to dial control conn 10.16.9.2:29042: EOF
01:58:48.341 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable to dial control conn 10.16.9.2:29042: EOF
01:58:48.342 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable to dial control conn 10.16.9.2:29042: EOF
01:58:48.640 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.640 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.640 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.640 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.640 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.641 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.641 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.641 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.642 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:59:58.481 app-foo plan-service-worker 2023/02/22 07:59:58 gocql: unable setup control conn 10.16.27.3:29042: Unexpected persistence error: No local service found for tenant 55555555-27f9-4540-8f42-db0faa162e9d
01:59:58.483 app-foo plan-service-worker 2023/02/22 07:59:58 gocql: unable setup control conn 10.16.27.3:29042: Unexpected persistence error: No local service found for tenant 55555555-27f9-4540-8f42-db0faa162e9d
02:06:15.441 app-foo plan-service-worker 2023/02/22 08:06:15 gocql: unable setup control conn 10.16.17.4:29042: Unexpected persistence error: No local service found for tenant 55555555-27f9-4540-8f42-db0faa162e9d
02:06:15.446 app-foo plan-service-worker 2023/02/22 08:06:15 gocql: unable setup control conn 10.16.17.4:29042: Unexpected persistence error: No local service found for tenant 55555555-27f9-4540-8f42-db0faa162e9d
02:06:15.446 app-foo plan-service-worker 2023/02/22 08:06:15 gocql: unable setup control conn 10.16.17.4:29
```

---

If you are having connectivity related issues please share the following additional information

### Describe your Cassandra cluster
please provide the following information

- output of `nodetool status`
- output of `SELECT peer, rpc_address FROM system.peers`
- rebuild your application with the `gocql_debug` tag and post the output

We don't have nodetool status in Astra. I'll rebuild with gocql_debug and provide data when the issue recurs or we have a chance to test.

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Reproduce the rolling restart against recycled Kubernetes nodes, rebuild the application with the gocql_debug tag, and inspect the reconnect attempts alongside nodetool status and system.peers output where available. Done means the driver attempts connections using the recycled nodes' current IP addresses and reconnects successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.