apache / apache/cassandra-gocql-driver

Add an interface to refresh contact points

Open
#1,681 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.7k
Forks
658
PR merge metrics
No merged PRs in 30d

Description

## Problem statement

After the introduction of the `HostDialer` interface (https://github.com/gocql/gocql/pull/1629), we have successfully created [a library that leverages this interface to use gocql to connect to Datastax Astra](https://github.com/datastax/gocql-astra) (original issue was https://github.com/gocql/gocql/pull/1487).

However we are seeing some errors during operations that cause the full set of C* hosts to change. During these operations new hosts are added to the C* cluster topology and old hosts are removed. I've troubleshooted these issues and I have found that currently `gocql` might drop some topology events if a status event is received roughly at the same time for the same host and I've also found that it is possible for the control connection to reconnect but not trigger a ring refresh which is an issue because during the reconnection process some events might be missed. Since these issues can affect any user of the driver regardless of their C* deployment type, I've submitted a separate PR to fix these (#1680).

Another issue that we discovered is that if, for any reason, the topology events are not correctly sent by the server (e.g. the server nodes crash instead of gracefully restarting/shutting down), then it is possible for gocql to be "stuck" with a list of hosts that are not valid anymore so it is unable to reconnect until the user forcefully closes the session/cluster and creates a new one.

## Proposal

**We are interested in adding an interface to gocql that would allow client libraries to "refresh" the contact points when `gocql` fails to reconnect the control connection**. I was thinking that it could be something like:

```
type ContactPointProvider interface {
Get() ([]*HostInfo, error)
}
```

Using this interface, a client library could provide an implementation that returns a list of `HostInfo` objects that would then be passed to the `HostDialer` (e.g. the `HostDialer` in `gocql-astra` uses host IDs as server names in TLS). In the specific case of `gocql-astra`, these hosts would have a `0.0.0.0` ip address because the IP address is only looked up via system tables during a ring refresh operation but `gocql-astra` is already calling `NewCluster("0.0.0.0")` and it works fine.

Alternatively, we could just use the same `string` type that is used for the `hosts` parameter in the `NewCluster()` function:
```
type ContactPointProvider interface {
Get() ([]string, error)
}
```
`gocql-astra`'s custom implementation would return a `0.0.0.0` string and `gocql` would then build a `HostInfo` object with empty host id and our custom `HostDialer` could then see that the provided `HostInfo` has no Host ID and the behavior would be the same as the one during session initialization (retrieve host IDs from Astra and use one of them as server name).

Other libraries and users in general would also be able to leverage this interface to provide a new set of contact points which might be useful in k8s and other types of "dynamic" environments. They can return a freshly resolved set of IP addresses or a new set of hostnames that `gocql` will then resolve.

This provider would be called by `gocql` during a control connection reconnection, after all current hosts have been attempted unsuccessfully. Then, `gocql` would use the new set of contact points to attempt to reconnect the control connection.

----

@martin-sucha Would you be willing to consider this proposal and review my implementation of it? Your work allowed us to create `gocql-astra` in the first place and this particular proposal could help not only us but also other users that are facing similar issues with k8s based deployments of C*.

This proposal is not related to #1680 but the goal of both #1680 and this proposal are the same: making the driver more robust in environments where C* topology changes happen frequently. Also, this proposal alone will not be enough if the control connection does not perform a ring refresh after a successful reconnection so merging #1680 would also be required in order for this proposal to be effective.

Contributor guide

Open the contributing guide

Research direction

Start by reading the HostDialer interface, NewCluster, and the control-connection reconnection and ring-refresh paths. Compare the proposed HostInfo and string ContactPointProvider alternatives, then determine how refreshed contact points are passed to HostDialer after current hosts fail. Done means the selected interface is integrated into reconnection behavior with tests covering refreshed contact points.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, go
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.