apache / apache/cassandra-gocql-driver
Speculative execution returning error to user when all replicas are taken by other executions
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 658
- PR merge metrics
- No merged PRs in 30d
Description
### What version of Cassandra are you using?
Scylla 5.1.12
### What version of Gocql are you using?
Latest master
### What version of Go are you using?
1.20
### What did you do?
In a 2 node cluster on a keyspace with replication factor = 2, I'm running queries with speculative execution and consistency = one while one of the nodes goes down.
### What did you expect to see?
The query execution should finish successfully, since there is a node available and consistency is set to one.
### What did you see instead?
When the main query execution takes a little longer and speculative retry is triggered, the speculative retry flow returns a `ErrNoConnections` and this error is returned to the app.
This is probably because the driver code does not wait for the main execution to return a response (either error or results) and returns the speculative execution's error. The speculative execution cannot get a replica to query because there is only a single one available and it is being used by the main execution.
Being more specific, in [this snippet](https://github.com/gocql/gocql/blob/master/query_executor.go#L92-L98) the speculative execution publishes an `Iter` with `err = ErrNoConnections` in the results channel.
One possible solution is to wait for the main execution to finish when the speculative execution returns an error (specially something like `ErrNoConnections`, for other errors it might make sense to forward to the app).
A possibly related issue is https://github.com/gocql/gocql/issues/1530
This is easily reproducible by setting `TimeoutDelay: 1` in the speculative execution config
Contributor guide
Research direction
Start in query_executor.go around lines 92-98, then reproduce the two-node Scylla 5.1.12 scenario with replication factor 2, consistency one, and speculative execution TimeoutDelay: 1. Compare the main execution with the speculative result when ErrNoConnections occurs, and review related issue #1530. Done means the available replica can complete the query successfully instead of exposing the speculative error to the application.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cassandra, go
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100