Performance of "skip"
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 9
Description
[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ )
## Description
I have a large Couchdb database with about 2 GB and I also use attachments to the documents.
[NOTE]: # ( Describe the problem you're encountering. )
[TIP]: # ( Do NOT give us access or passwords to your actual CouchDB! )
## Steps to Reproduce
time curl -X GET URL/_all_docs?include_docs=true&attachments=true&limit=1&skip=1
=> Requires almost no time
time curl -X GET URL/_all_docs?include_docs=true&attachments=true&limit=1&skip=500
=> Requires 16 seconds
time curl -X GET URL/_all_docs?include_docs=true&attachments=true&limit=1&skip=1000
=> Requires 32 seconds
time curl -X GET URL/_all_docs?include_docs=true&attachments=true&limit=1&skip=2000
=> Requires 62 seconds
When looking at this, please note that I use a limit=1 in all cases to get similar results. The sizes of the documents and attachments are about the same in all documents.
[NOTE]: # ( Include commands to reproduce, if possible. curl is preferred. )
## Expected Behaviour
I would expect the skip command to cause the data to be fetched only from the skip position. And this should be fast, regardless of the position of the document.
It seems that a skip command, however, would internally perform a full query of all data, and then throw away the data that should not be delivered (skipped). This leads to unnecessarily time-consuming queries for large databases. Especially when fetching documents that have a higher index.
[NOTE]: # ( Tell us what you expected to happen. )
## Your Environment
{
"couchdb": "Welcome",
"version": "3.1.1",
"git_sha": "ce596c65d",
"uuid": "a6101edd1ffd110ace6ed3e32a80417f",
"features": [
"access-ready",
"partitioned",
"pluggable-storage-engines",
"reshard",
"scheduler"
],
"vendor": {
"name": "The Apache Software Foundation"
}
}
[TIP]: # ( Include as many relevant details about your environment as possible. )
[TIP]: # ( You can paste the output of curl http://YOUR-COUCHDB:5984/ here. )
* CouchDB version used: 3.1.1
* Browser name and version: curl
* Operating system and version: linux / docker
## Additional Context
[TIP]: # ( Add any other context about the problem here. )
Contributor guide
Research direction
Start with the _all_docs HTTP endpoint and its skip, limit, include_docs, and attachments parameters, using the curl commands in the report to reproduce the increasing response time. Done means determining whether the skip behavior can avoid processing earlier documents and validating the performance change on a large database.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, erlang, linux
- Domain
- api, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100