apache / apache/cassandra-gocql-driver
Rolling restart will not attempt reconnect to recycled nodes (ip's are changing on recycled nodes, kubernetes pods)
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 658
- PR merge metrics
- No merged PRs in 30d
Description
Please answer these questions before submitting your issue. Thanks!
### What version of Cassandra are you using?
DataStax Astra - 6.8 DSE
### What version of Gocql are you using?
1.20
### What version of Go are you using?
### What did you do?
When rolling restart of the cluster, the application is failing to reconnect.
### What did you expect to see?
Reconnect to the nodes that have been recycled and are now up.
### What did you see instead?
Here's the log when the connection is failing:
```
01:58:48.338 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable to dial control conn 10.16.9.2:29042: EOF
01:58:48.341 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable to dial control conn 10.16.9.2:29042: EOF
01:58:48.342 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable to dial control conn 10.16.9.2:29042: EOF
01:58:48.640 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.640 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.640 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.640 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.640 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.641 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.641 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.641 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:58:48.642 app-foo plan-service-worker 2023/02/22 07:58:48 gocql: unable setup control conn 10.16.9.2:29042: EOF
01:59:58.481 app-foo plan-service-worker 2023/02/22 07:59:58 gocql: unable setup control conn 10.16.27.3:29042: Unexpected persistence error: No local service found for tenant 55555555-27f9-4540-8f42-db0faa162e9d
01:59:58.483 app-foo plan-service-worker 2023/02/22 07:59:58 gocql: unable setup control conn 10.16.27.3:29042: Unexpected persistence error: No local service found for tenant 55555555-27f9-4540-8f42-db0faa162e9d
02:06:15.441 app-foo plan-service-worker 2023/02/22 08:06:15 gocql: unable setup control conn 10.16.17.4:29042: Unexpected persistence error: No local service found for tenant 55555555-27f9-4540-8f42-db0faa162e9d
02:06:15.446 app-foo plan-service-worker 2023/02/22 08:06:15 gocql: unable setup control conn 10.16.17.4:29042: Unexpected persistence error: No local service found for tenant 55555555-27f9-4540-8f42-db0faa162e9d
02:06:15.446 app-foo plan-service-worker 2023/02/22 08:06:15 gocql: unable setup control conn 10.16.17.4:29
```
---
If you are having connectivity related issues please share the following additional information
### Describe your Cassandra cluster
please provide the following information
- output of `nodetool status`
- output of `SELECT peer, rpc_address FROM system.peers`
- rebuild your application with the `gocql_debug` tag and post the output
We don't have nodetool status in Astra. I'll rebuild with gocql_debug and provide data when the issue recurs or we have a chance to test.
Contributor guide
Research direction
No source file or test is named. Reproduce the rolling restart against recycled Kubernetes nodes, rebuild the application with the gocql_debug tag, and inspect the reconnect attempts alongside nodetool status and system.peers output where available. Done means the driver attempts connections using the recycled nodes' current IP addresses and reconnects successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100