apache / apache/cassandra-gocql-driver

query retries rarely cause retries because hostpoolpolicy aborts query execution

Open
#812 8 comments 4 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.7k
Forks
658
PR merge metrics
No merged PRs in 30d

Description

So I don't know if this works as designed or not. At least I found things counter-intuitive and spent quite some time troubleshooting what's going on.

In my [client app](https://github.com/raintank/metrictank) I've implemented a config option to enable query retries, which just enables a SimpleRetryPolicy within gocql.
I then use toxiproxy to artifically slow down cassandra, somewhat randomly, so that some queries won't be slowed down much, while other are slower than the timeout. Note that in my testing i just use 1 cassandra instance.

As I've tested this, with higher and higher query retry settings, i notice that only some queries are retried, and only a limited amount, definitely not to the amount permitted by the SimpleRetryPolicy.
And so they still return timeout errors, even though could have been retried more often and returned a valid result instead.

The reason is that `queryExecutor.executeQuery(qry ExecutableQuery)` stops attempting a query when the hostpool Pick() function returns nil, instead of a valid host.

I thought I had read somewhere that when a hostpool becomes empty (e.g. all hosts timed out), then they would all be re-added again, so that the query could be retried again. But this does not seem to be case.

I verified this with both the round-robin host selection policy as well tokenAware with a simple hostpool backend. (I know tokenaware doesn't effect anything here because i have 1 node, presumably this is equivalent to using hostpool-simple directly)

with NumRetries 3/4/5/6/... and round-robin i saw that many queries were only tried twice (retried once) and timed out, and with tokenaware+hostpool-simple only 2 were tried twice, resulting in a timeout and "too many timeouts" errors, all queries afterwards were only tried once.

is this normal?

Contributor guide

Open the contributing guide

Research direction

Start with queryExecutor.executeQuery(qry ExecutableQuery) and inspect how the hostpool Pick() result interacts with SimpleRetryPolicy. Reproduce the behavior using a one-node Cassandra hostpool, round-robin or token-aware selection, and toxiproxy-induced delays. Done means the retry and host exhaustion behavior is understood and the issue's expected behavior is documented or confirmed.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, go
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.