apache / apache/cassandra-gocql-driver

Speculative execution returning error to user when all replicas are taken by other executions

Open
#1,707 2 comments 3 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.7k
Forks
658
PR merge metrics
No merged PRs in 30d

Description

### What version of Cassandra are you using?
Scylla 5.1.12

### What version of Gocql are you using?
Latest master

### What version of Go are you using?
1.20

### What did you do?
In a 2 node cluster on a keyspace with replication factor = 2, I'm running queries with speculative execution and consistency = one while one of the nodes goes down.

### What did you expect to see?
The query execution should finish successfully, since there is a node available and consistency is set to one.

### What did you see instead?
When the main query execution takes a little longer and speculative retry is triggered, the speculative retry flow returns a `ErrNoConnections` and this error is returned to the app.

This is probably because the driver code does not wait for the main execution to return a response (either error or results) and returns the speculative execution's error. The speculative execution cannot get a replica to query because there is only a single one available and it is being used by the main execution.

Being more specific, in [this snippet](https://github.com/gocql/gocql/blob/master/query_executor.go#L92-L98) the speculative execution publishes an `Iter` with `err = ErrNoConnections` in the results channel.

One possible solution is to wait for the main execution to finish when the speculative execution returns an error (specially something like `ErrNoConnections`, for other errors it might make sense to forward to the app).

A possibly related issue is https://github.com/gocql/gocql/issues/1530

This is easily reproducible by setting `TimeoutDelay: 1` in the speculative execution config

Contributor guide

Open the contributing guide

Research direction

Start in query_executor.go around lines 92-98, then reproduce the two-node Scylla 5.1.12 scenario with replication factor 2, consistency one, and speculative execution TimeoutDelay: 1. Compare the main execution with the speculative result when ErrNoConnections occurs, and review related issue #1530. Done means the available replica can complete the query successfully instead of exposing the speculative error to the application.

Written by the indexing model from the issue text.

Assessment

Tech stack
cassandra, go
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.