apache / apache/couchdb

Performance of "skip"

Open
#3,562 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ )

## Description

I have a large Couchdb database with about 2 GB and I also use attachments to the documents.

[NOTE]: # ( Describe the problem you're encountering. )
[TIP]: # ( Do NOT give us access or passwords to your actual CouchDB! )

## Steps to Reproduce

time curl -X GET URL/_all_docs?include_docs=true&attachments=true&limit=1&skip=1
=> Requires almost no time

time curl -X GET URL/_all_docs?include_docs=true&attachments=true&limit=1&skip=500
=> Requires 16 seconds

time curl -X GET URL/_all_docs?include_docs=true&attachments=true&limit=1&skip=1000
=> Requires 32 seconds

time curl -X GET URL/_all_docs?include_docs=true&attachments=true&limit=1&skip=2000
=> Requires 62 seconds

When looking at this, please note that I use a limit=1 in all cases to get similar results. The sizes of the documents and attachments are about the same in all documents.

[NOTE]: # ( Include commands to reproduce, if possible. curl is preferred. )

## Expected Behaviour

I would expect the skip command to cause the data to be fetched only from the skip position. And this should be fast, regardless of the position of the document.

It seems that a skip command, however, would internally perform a full query of all data, and then throw away the data that should not be delivered (skipped). This leads to unnecessarily time-consuming queries for large databases. Especially when fetching documents that have a higher index.

[NOTE]: # ( Tell us what you expected to happen. )

## Your Environment

{
"couchdb": "Welcome",
"version": "3.1.1",
"git_sha": "ce596c65d",
"uuid": "a6101edd1ffd110ace6ed3e32a80417f",
"features": [
"access-ready",
"partitioned",
"pluggable-storage-engines",
"reshard",
"scheduler"
],
"vendor": {
"name": "The Apache Software Foundation"
}
}

[TIP]: # ( Include as many relevant details about your environment as possible. )
[TIP]: # ( You can paste the output of curl http://YOUR-COUCHDB:5984/ here. )

* CouchDB version used: 3.1.1
* Browser name and version: curl
* Operating system and version: linux / docker

## Additional Context

[TIP]: # ( Add any other context about the problem here. )

Contributor guide

Open the contributing guide

Research direction

Start with the _all_docs HTTP endpoint and its skip, limit, include_docs, and attachments parameters, using the curl commands in the report to reproduce the increasing response time. Done means determining whether the skip behavior can avoid processing earlier documents and validating the performance change on a large database.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, erlang, linux
Domain
api, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.