apache / apache/cassandra-gocql-driver
query retries rarely cause retries because hostpoolpolicy aborts query execution
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 658
- PR merge metrics
- No merged PRs in 30d
Description
So I don't know if this works as designed or not. At least I found things counter-intuitive and spent quite some time troubleshooting what's going on.
In my [client app](https://github.com/raintank/metrictank) I've implemented a config option to enable query retries, which just enables a SimpleRetryPolicy within gocql.
I then use toxiproxy to artifically slow down cassandra, somewhat randomly, so that some queries won't be slowed down much, while other are slower than the timeout. Note that in my testing i just use 1 cassandra instance.
As I've tested this, with higher and higher query retry settings, i notice that only some queries are retried, and only a limited amount, definitely not to the amount permitted by the SimpleRetryPolicy.
And so they still return timeout errors, even though could have been retried more often and returned a valid result instead.
The reason is that `queryExecutor.executeQuery(qry ExecutableQuery)` stops attempting a query when the hostpool Pick() function returns nil, instead of a valid host.
I thought I had read somewhere that when a hostpool becomes empty (e.g. all hosts timed out), then they would all be re-added again, so that the query could be retried again. But this does not seem to be case.
I verified this with both the round-robin host selection policy as well tokenAware with a simple hostpool backend. (I know tokenaware doesn't effect anything here because i have 1 node, presumably this is equivalent to using hostpool-simple directly)
with NumRetries 3/4/5/6/... and round-robin i saw that many queries were only tried twice (retried once) and timed out, and with tokenaware+hostpool-simple only 2 were tried twice, resulting in a timeout and "too many timeouts" errors, all queries afterwards were only tried once.
is this normal?
Contributor guide
Research direction
Start with queryExecutor.executeQuery(qry ExecutableQuery) and inspect how the hostpool Pick() result interacts with SimpleRetryPolicy. Reproduce the behavior using a one-node Cassandra hostpool, round-robin or token-aware selection, and toxiproxy-induced delays. Done means the retry and host exhaustion behavior is understood and the issue's expected behavior is documented or confirmed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cassandra, go
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100