Implement quorum limit for search endpoints
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
## Summary
Many endpoints respond only when a quorum of replicas have responded with what the actual response should be, however not all endpoints behave this way. As @rnewson said in Slack:
> the _view, _find and _search endpoints read one copy of each shard range to form the response, there is no option to consult multiple copies, the code simply hasn't been written.
This can cause bugs in applications that make successive queries where these may be ultimately handled by different nodes.
## Desired Behaviour
All endpoints to behave consistently using the default quorum which can be overridden (eg: with `r` and `w` query parameters). This makes the behaviour of endpoints consistent and therefore easier to develop against, but also means subsequent queries can be guaranteed to work given appropriate values for the quorum.
## Possible Solution
Have these endpoints read all copies of the shard and respond when a quorum is achieved.
## Additional context
We're upgrading from supporting single node to clustered operation and certain tests in our integration suite are failing too often.
Some workarounds we have considered include, using `w=` everywhere, setting replicas to 1, and now investigating using sticky sessions so subsequent requests from the same client are handled by the same node. These have obvious drawbacks and none have been found (yet) to be 100% reliable.
Contributor guide
Research direction
Start by tracing the _view, _find, and _search endpoint implementations and the integration tests that fail during clustered operation. Determine how replica reads and quorum parameters are handled elsewhere; done means these endpoints consult shard copies and honor the default quorum plus r and w overrides consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- erlang
- Domain
- api, database, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100