apache / apache/cassandra-gocql-driver
New hosts connection details are not updated
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 658
- PR merge metrics
- No merged PRs in 30d
Description
### What version of Cassandra are you using?
datastax 6.8.28
### What version of Gocql are you using?
`1.3.1`
### What version of Go are you using?
`go1.19.2`
### What did you do?
cassandra is deployed and managed using k8ssandra-oprerator
certain pods of cassandra were replaced due to a kubernetes node-down event.
### What did you expect to see?
gocql should gracefully handle changes in topology
### What did you see instead?
when cassandra nodes (pods) are replaced, their IPs change, gocql client does not update hostinfo with new IPs
---
If you are having connectivity related issues please share the following additional information
### Describe your Cassandra cluster
please provide the following information
- output of `nodetool status`
all nodes are in `UN` state
- output of `SELECT peer, rpc_address FROM system.peers`
- rebuild your application with the `gocql_debug` tag and post the output
```
time="2022-12-23T07:00:55Z" level=debug msg="gocql: Session.handleNodeDown: 10.244.101.39:9042\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:00Z" level=error msg="error connecting: dial tcp 10.244.125.76:9042: i/o timeout (hostinfo=[HostInfo hostname=\"10.244.125.76\" connectAddress=\"10.244.125.76\" peer=\"10.244.125.76\" rpc_address=\"10.244.125.76\" broadcast_address=\"\" preferred_ip=\"\" connect_addr=\"10.244.125.76\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter-id\" rack=\"rack-2\" host_id=\"59b3e71a-319e-4438-8104-67e536c1977f\" version=\"v4.0.0\" state=DOWN num_tokens=8])" Id=2
time="2022-12-23T07:01:00Z" level=debug msg="gocql: connection failed \"10.244.125.76\": dial tcp 10.244.125.76:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:06Z" level=error msg="error connecting: dial tcp 10.244.125.76:9042: i/o timeout (hostinfo=[HostInfo hostname=\"10.244.125.76\" connectAddress=\"10.244.125.76\" peer=\"10.244.125.76\" rpc_address=\"10.244.125.76\" broadcast_address=\"\" preferred_ip=\"\" connect_addr=\"10.244.125.76\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter-id\" rack=\"rack-2\" host_id=\"59b3e71a-319e-4438-8104-67e536c1977f\" version=\"v4.0.0\" state=DOWN num_tokens=8])" Id=2
time="2022-12-23T07:01:06Z" level=debug msg="gocql: connection failed \"10.244.125.76\": dial tcp 10.244.125.76:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:12Z" level=error msg="error connecting: dial tcp 10.244.125.76:9042: i/o timeout (hostinfo=[HostInfo hostname=\"10.244.125.76\" connectAddress=\"10.244.125.76\" peer=\"10.244.125.76\" rpc_address=\"10.244.125.76\" broadcast_address=\"\" preferred_ip=\"\" connect_addr=\"10.244.125.76\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter-id\" rack=\"rack-2\" host_id=\"59b3e71a-319e-4438-8104-67e536c1977f\" version=\"v4.0.0\" state=DOWN num_tokens=8])" Id=2
time="2022-12-23T07:01:12Z" level=debug msg="gocql: connection failed \"10.244.125.76\": dial tcp 10.244.125.76:9042: i/o timeout, reconnecting with *gocql.ConstantReconnectionPolicy\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:13Z" level=debug msg="gocql: unable to dial \"[HostInfo hostname=\\\"10.244.125.76\\\" connectAddress=\\\"10.244.125.76\\\" peer=\\\"10.244.125.76\\\" rpc_address=\\\"10.244.125.76\\\" broadcast_address=\\\"\\\" preferred_ip=\\\"\\\" connect_addr=\\\"10.244.125.76\\\" connect_addr_source=\\\"connect_address\\\" port=9042 data_centre=\\\"datacenter-id\\\" rack=\\\"rack-2\\\" host_id=\\\"59b3e71a-319e-4438-8104-67e536c1977f\\\" version=\\\"v4.0.0\\\" state=DOWN num_tokens=8]\": dial tcp 10.244.125.76:9042: i/o timeout\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:13Z" level=debug msg="gocql: filling stopped \"10.244.125.76\": dial tcp 10.244.125.76:9042: i/o timeout\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:13Z" level=debug msg="gocql: conns of pool after stopped \"10.244.125.76\": 0\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:13Z" level=debug msg="gocql: Session.handleNodeDown: 10.244.125.76:9042\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:22Z" level=debug msg="gocql: pool connection error \"10.244.111.100:9042\": read tcp 10.244.113.56:60504->10.244.111.100:9042: read: connection reset by peer\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:22Z" level=error msg="error in query: read tcp 10.244.113.56:60504->10.244.111.100:9042: read: connection reset by peer" Id=1750
time="2022-12-23T07:01:22Z" level=error msg="error in query: gocql: connection closed waiting for response" Id=3934
time="2022-12-23T07:01:22Z" level=debug msg="gocql: pool connection error \"10.244.111.100:9042\": read tcp 10.244.113.56:60492->10.244.111.100:9042: read: connection reset by peer\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:22Z" level=error msg="error in query: read tcp 10.244.113.56:60492->10.244.111.100:9042: read: connection reset by peer" Id=1906
time="2022-12-23T07:01:22Z" level=error msg="error in query: read tcp 10.244.113.56:60504->10.244.111.100:9042: read: connection reset by peer" Id=2833
time="2022-12-23T07:01:22Z" level=error msg="error in query: gocql: no hosts available in the pool" Id=1
time="2022-12-23T07:01:22Z" level=error msg="error in query: read tcp 10.244.113.56:60504->10.244.111.100:9042: read: connection reset by peer" Id=1439
time="2022-12-23T07:01:22Z" level=error msg="error in query: read tcp 10.244.113.56:60492->10.244.111.100:9042: read: connection reset by peer" Id=3729
time="2022-12-23T07:01:22Z" level=error msg="error in query: gocql: no hosts available in the pool" Id=1246
time="2022-12-23T07:01:22Z" level=error msg="error connecting: dial tcp 10.244.111.100:9042: connect: connection refused (hostinfo=[HostInfo hostname=\"10.244.111.100\" connectAddress=\"10.244.111.100\" peer=\"\" rpc_address=\"10.244.111.100\" broadcast_address=\"10.244.111.100\" preferred_ip=\"\" connect_addr=\"10.244.111.100\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter-id\" rack=\"rack-3\" host_id=\"52fbc88e-8f99-4f42-9882-a16a6018527a\" version=\"v4.0.0\" state=UP num_tokens=8])" Id=2
time="2022-12-23T07:01:22Z" level=debug msg="gocql: unable to dial control conn 10.244.111.100:9042: dial tcp 10.244.111.100:9042: connect: connection refused\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:22Z" level=error msg="error in query: gocql: no hosts available in the pool" Id=3871
time="2022-12-23T07:01:22Z" level=error msg="error connecting: dial tcp 10.244.111.100:9042: connect: connection refused (hostinfo=[HostInfo hostname=\"10.244.111.100\" connectAddress=\"10.244.111.100\" peer=\"\" rpc_address=\"10.244.111.100\" broadcast_address=\"10.244.111.100\" preferred_ip=\"\" connect_addr=\"10.244.111.100\" connect_addr_source=\"connect_address\" port=9042 data_centre=\"datacenter-id\" rack=\"rack-3\" host_id=\"52fbc88e-8f99-4f42-9882-a16a6018527a\" version=\"v4.0.0\" state=UP num_tokens=8])" Id=2
time="2022-12-23T07:01:22Z" level=debug msg="gocql: unable to dial \"[HostInfo hostname=\\\"10.244.111.100\\\" connectAddress=\\\"10.244.111.100\\\" peer=\\\"\\\" rpc_address=\\\"10.244.111.100\\\" broadcast_address=\\\"10.244.111.100\\\" preferred_ip=\\\"\\\" connect_addr=\\\"10.244.111.100\\\" connect_addr_source=\\\"connect_address\\\" port=9042 data_centre=\\\"datacenter-id\\\" rack=\\\"rack-3\\\" host_id=\\\"52fbc88e-8f99-4f42-9882-a16a6018527a\\\" version=\\\"v4.0.0\\\" state=UP num_tokens=8]\": dial tcp 10.244.111.100:9042: connect: connection refused\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:22Z" level=debug msg="gocql: filling stopped \"10.244.111.100\": dial tcp 10.244.111.100:9042: connect: connection refused\n" sess-key="database-dummy.domain.com:9042"
time="2022-12-23T07:01:22Z" level=error msg="error in query: gocql: no hosts available in the pool" Id=1159
```
Contributor guide
Research direction
Start with the reported Cassandra 4.0 and GoCQL 1.3.1 setup, reproducing replacement of Cassandra pods and reviewing the gocql_debug output. Trace Session.handleNodeDown and the HostInfo values shown in the logs. Done means host details follow new pod IPs after topology changes and the client can continue connecting without stale hosts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cassandra, go
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100